davidmalcolm / gcc-analyzer-integration-tests

6 stars 1 forks source link

Add performance testing #5

Open davidmalcolm opened 1 year ago

davidmalcolm commented 1 year ago

We should probably check how much time taken is by the analyzer.

My goal is that the analyzer should "merely" double the compile time.

Perhaps express time taken by analyzer as a multiple of time taken by the rest of the compile; group files by the multiple?

Probably should add absolute and comparison report scripts for this

I've filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109361 with GCC to track adding profiling information to the SARIF files

davidmalcolm commented 1 year ago

Some initial results, showing 50 slowest compiles, and what analyzer item took longest for each one:

50 slowest analyses:
  qemu-7.2.0/build/target_hexagon_genptr.c.c.sarif: 791.17s (13.19m) out of 881.16s (89.79%; slowdown = 9.79)
    without -fanalyzer would be: 89.99s
    processing worklist: 784.51s
  qemu-7.2.0/build/target_arm_translate-sve.c.c.sarif: 521.77s (8.70m) out of 590.54s (88.35%; slowdown = 8.59)
    without -fanalyzer would be: 68.77s
    processing worklist: 517.21s
  ImageMagick-7.1.0-57/libMagickWand_7_Q16HDRI_la-mogrify.c.sarif: 490.77s (8.18m) out of 506.07s (96.98%; slowdown = 33.08)
    without -fanalyzer would be: 15.30s
    processing worklist: 480.40s
  qemu-7.2.0/build/target_arm_sve_helper.c.c.sarif: 325.70s (5.43m) out of 482.92s (67.44%; slowdown = 3.07)
    without -fanalyzer would be: 157.22s
    processing worklist: 314.77s
  qemu-7.2.0/build/target_ppc_translate.c.c.sarif: 310.87s (5.18m) out of 364.86s (85.20%; slowdown = 6.76)
    without -fanalyzer would be: 53.99s
    processing worklist: 306.30s
  qemu-7.2.0/build/target_riscv_translate.c.c.sarif: 284.22s (4.74m) out of 335.95s (84.60%; slowdown = 6.49)
    without -fanalyzer would be: 51.73s
    processing worklist: 280.06s
  ImageMagick-7.1.0-57/libMagickCore_7_Q16HDRI_la-display.c.sarif: 260.33s (4.34m) out of 265.95s (97.89%; slowdown = 47.32)
    without -fanalyzer would be: 5.62s
    processing worklist: 255.50s
  qemu-7.2.0/build/disas_cris.c.c.sarif: 185.09s (3.08m) out of 187.21s (98.87%; slowdown = 88.31)
    without -fanalyzer would be: 2.12s
    processing worklist: 184.70s
  qemu-7.2.0/build/target_i386_tcg_translate.c.c.sarif: 175.66s (2.93m) out of 200.81s (87.48%; slowdown = 7.98)
    without -fanalyzer would be: 25.15s
    processing worklist: 172.97s
  ImageMagick-7.1.0-57/MagickCore_libMagickCore_7_Q16HDRI_la-png.c.sarif: 164.61s (2.74m) out of 170.49s (96.55%; slowdown = 28.99)
    without -fanalyzer would be: 5.88s
    processing worklist: 147.96s
  pcre2-10.42/libpcre2_8_la-pcre2_match.c.sarif: 154.88s (2.58m) out of 159.23s (97.27%; slowdown = 36.60)
    without -fanalyzer would be: 4.35s
    processing worklist: 85.94s
  qemu-7.2.0/build/target_riscv_vector_helper.c.c.sarif: 129.62s (2.16m) out of 193.74s (66.90%; slowdown = 3.02)
    without -fanalyzer would be: 64.12s
    processing worklist: 125.56s
  qemu-7.2.0/build/target_arm_translate-a64.c.c.sarif: 128.68s (2.14m) out of 166.64s (77.22%; slowdown = 4.39)
    without -fanalyzer would be: 37.96s
    processing worklist: 124.02s
  qemu-7.2.0/build/target_mips_tcg_translate.c.c.sarif: 120.55s (2.01m) out of 166.01s (72.62%; slowdown = 3.65)
    without -fanalyzer would be: 45.46s
    processing worklist: 118.03s
  ImageMagick-7.1.0-57/libMagickWand_7_Q16HDRI_la-display.c.sarif: 120.27s (2.00m) out of 125.42s (95.89%; slowdown = 24.35)
    without -fanalyzer would be: 5.15s
    processing worklist: 117.80s
  qemu-7.2.0/build/tcg_tcg.c.c.sarif: 116.82s (1.95m) out of 125.27s (93.25%; slowdown = 14.82)
    without -fanalyzer would be: 8.45s
    shortest paths: 72.41s
  git-2.39.0/pack-objects.c.sarif: 106.79s (1.78m) out of 108.81s (98.14%; slowdown = 53.87)
    without -fanalyzer would be: 2.02s
    processing worklist: 105.55s
  qemu-7.2.0/build/target_hexagon_decode.c.c.sarif: 105.76s (1.76m) out of 130.19s (81.24%; slowdown = 5.33)
    without -fanalyzer would be: 24.43s
    processing worklist: 104.21s
  git-2.39.0/sequencer.c.sarif: 102.54s (1.71m) out of 105.56s (97.14%; slowdown = 34.95)
    without -fanalyzer would be: 3.02s
    processing worklist: 89.03s
  qemu-7.2.0/build/target_arm_translate.c.c.sarif: 98.25s (1.64m) out of 137.20s (71.61%; slowdown = 3.52)
    without -fanalyzer would be: 38.95s
    processing worklist: 96.06s
  qemu-7.2.0/build/target_s390x_tcg_translate.c.c.sarif: 94.99s (1.58m) out of 126.86s (74.88%; slowdown = 3.98)
    without -fanalyzer would be: 31.87s
    processing worklist: 93.27s
  coreutils-9.1/factor.c.sarif: 94.81s (1.58m) out of 95.75s (99.02%; slowdown = 101.86)
    without -fanalyzer would be: 0.94s
    processing worklist: 94.49s
  ImageMagick-7.1.0-57/libMagickWand_7_Q16HDRI_la-operation.c.sarif: 89.61s (1.49m) out of 92.42s (96.96%; slowdown = 32.89)
    without -fanalyzer would be: 2.81s
    processing worklist: 70.77s
  haproxy-2.7.1/http_ana.c.sarif: 86.85s (1.45m) out of 94.85s (91.57%; slowdown = 11.86)
    without -fanalyzer would be: 8.00s
    processing worklist: 85.82s
  qemu-7.2.0/build/hw_nvme_ctrl.c.c.sarif: 84.64s (1.41m) out of 103.87s (81.49%; slowdown = 5.40)
    without -fanalyzer would be: 19.23s
    processing worklist: 38.66s
  git-2.39.0/merge-ort.c.sarif: 83.98s (1.40m) out of 85.83s (97.84%; slowdown = 46.39)
    without -fanalyzer would be: 1.85s
    processing worklist: 76.79s
  git-2.39.0/merge-recursive.c.sarif: 78.40s (1.31m) out of 80.09s (97.89%; slowdown = 47.39)
    without -fanalyzer would be: 1.69s
    processing worklist: 61.59s
  pcre2-10.42/libpcre2_8_la-pcre2_compile.c.sarif: 72.45s (1.21m) out of 74.08s (97.80%; slowdown = 45.45)
    without -fanalyzer would be: 1.63s
    processing worklist: 65.93s
  ImageMagick-7.1.0-57/libMagickWand_7_Q16HDRI_la-montage.c.sarif: 72.45s (1.21m) out of 77.86s (93.05%; slowdown = 14.39)
    without -fanalyzer would be: 5.41s
    processing worklist: 54.70s
  git-2.39.0/fast-import.c.sarif: 71.48s (1.19m) out of 73.19s (97.66%; slowdown = 42.80)
    without -fanalyzer would be: 1.71s
    processing worklist: 70.53s
  qemu-7.2.0/build/meson-generated_.._qapi_qapi-visit-block-core.c.c.sarif: 70.14s (1.17m) out of 83.51s (83.99%; slowdown = 6.25)
    without -fanalyzer would be: 13.37s
    processing worklist: 68.51s
  qemu-7.2.0/build/hw_intc_arm_gicv3_its.c.c.sarif: 69.49s (1.16m) out of 74.00s (93.91%; slowdown = 16.41)
    without -fanalyzer would be: 4.51s
    shortest paths: 35.88s
  ImageMagick-7.1.0-57/libMagickCore_7_Q16HDRI_la-widget.c.sarif: 68.26s (1.14m) out of 72.28s (94.44%; slowdown = 17.98)
    without -fanalyzer would be: 4.02s
    processing worklist: 65.02s
  pcre2-10.42/pcre2test-pcre2test.c.sarif: 67.69s (1.13m) out of 69.47s (97.44%; slowdown = 39.03)
    without -fanalyzer would be: 1.78s
    processing worklist: 66.29s
  ImageMagick-7.1.0-57/libMagickCore_7_Q16HDRI_la-draw.c.sarif: 66.88s (1.11m) out of 70.59s (94.74%; slowdown = 19.03)
    without -fanalyzer would be: 3.71s
    processing worklist: 56.27s
  qemu-7.2.0/build/fpu_softfloat.c.c.sarif: 64.36s (1.07m) out of 88.86s (72.43%; slowdown = 3.63)
    without -fanalyzer would be: 24.50s
    processing worklist: 62.04s
  qemu-7.2.0/build/block_qcow2-refcount.c.c.sarif: 63.62s (1.06m) out of 70.19s (90.64%; slowdown = 10.68)
    without -fanalyzer would be: 6.57s
    processing worklist: 62.75s
  coreutils-9.1/sort.c.sarif: 63.24s (1.05m) out of 64.74s (97.68%; slowdown = 43.16)
    without -fanalyzer would be: 1.50s
    processing worklist: 57.73s
  coreutils-9.1/libcoreutils_a-regex.c.sarif: 61.21s (1.02m) out of 64.23s (95.30%; slowdown = 21.27)
    without -fanalyzer would be: 3.02s
    processing worklist: 49.18s
  qemu-7.2.0/build/.._.._fpu_softfloat.c.c.sarif: 58.05s out of 81.26s (71.44%; slowdown = 3.50)
    without -fanalyzer would be: 23.21s
    processing worklist: 55.92s
  ImageMagick-7.1.0-57/MagickCore_libMagickCore_7_Q16HDRI_la-tiff.c.sarif: 57.24s out of 59.19s (96.71%; slowdown = 30.35)
    without -fanalyzer would be: 1.95s
    processing worklist: 55.68s
  qemu-7.2.0/build/target_arm_translate-mve.c.c.sarif: 56.57s out of 88.52s (63.91%; slowdown = 2.77)
    without -fanalyzer would be: 31.95s
    processing worklist: 54.99s
  qemu-7.2.0/build/softmmu_memory.c.c.sarif: 56.45s out of 64.82s (87.09%; slowdown = 7.74)
    without -fanalyzer would be: 8.37s
    processing worklist: 55.62s
  git-2.39.0/revision.c.sarif: 54.78s out of 56.77s (96.49%; slowdown = 28.53)
    without -fanalyzer would be: 1.99s
    processing worklist: 35.12s
  git-2.39.0/apply.c.sarif: 54.24s out of 56.50s (96.00%; slowdown = 25.00)
    without -fanalyzer would be: 2.26s
    processing worklist: 53.19s
  coreutils-9.1/copy.c.sarif: 54.21s out of 55.18s (98.24%; slowdown = 56.89)
    without -fanalyzer would be: 0.97s
    processing worklist: 33.22s
  haproxy-2.7.1/mux_h1.c.sarif: 53.98s out of 65.96s (81.84%; slowdown = 5.51)
    without -fanalyzer would be: 11.98s
    processing worklist: 48.89s
  qemu-7.2.0/build/target_tricore_translate.c.c.sarif: 52.00s out of 78.48s (66.26%; slowdown = 2.96)
    without -fanalyzer would be: 26.48s
    processing worklist: 50.46s
  qemu-7.2.0/build/disas_sh4.c.c.sarif: 50.02s out of 51.45s (97.22%; slowdown = 35.98)
    without -fanalyzer would be: 1.43s
    processing worklist: 49.95s
  git-2.39.0/object-file.c.sarif: 49.21s out of 50.86s (96.76%; slowdown = 30.82)
    without -fanalyzer would be: 1.65s
    processing worklist: 48.56s