search

nordlow / compiler-benchmark

Benchmarks compilation speeds of different combinations of languages and compilers.
MIT License
138 stars 18 forks source link

Execution time for Julia is reported incorrectly #16

Open ufechner7 opened 3 years ago

ufechner7 commented 3 years ago

If I execute:

./benchmark --languages=C,Julia

I get the following result:

| Lang-uage | Temp-lated | Check Time [us/fn] | Compile Time [us/fn] | Build Time [us/fn] | Run Time [us/fn] | Check RSS [kB/fn] | Build RSS [kB/fn] | Exec Version | Exec Path | 
| :-------: | ---------- | :----------------: | :------------------: | :----------------: | :--------------: | :---------------: | :---------------: | :----------: | :-------: | 
| C         | No         |    2.1 (best)      |    2.0 (best)        |    3.0 (best)      |    124 (best)    |    0.4 (best)     |    0.7 (best)     | 0.9.27       | tcc       | 
| C         | No         |   10.5 (5.1x)      |  290.8 (144.7x)      |  292.1 (95.8x)     |    611 (4.9x)    |    4.3 (10.0x)    |   22.4 (33.3x)    | 9.3.0        | gcc       | 
| C         | No         |   12.0 (5.8x)      |  296.7 (147.6x)      |  294.9 (96.7x)     |    158 (1.3x)    |    4.3 (9.9x)     |   22.4 (33.3x)    | 9.3.0        | gcc-9     | 
| C         | No         |    7.5 (3.7x)      |  282.0 (140.4x)      |  265.9 (87.2x)     |    147 (1.2x)    |    3.5 (8.1x)     |   21.6 (32.2x)    | 10.3.0       | gcc-10    | 
| C         | No         |   24.6 (12.0x)     |  132.9 (66.1x)       |  127.4 (41.8x)     |   1219 (9.8x)    |    6.2 (14.3x)    |   18.3 (27.2x)    | 10.0.0-4     | clang-10  | 
| Julia     | No         |    N/A             |    N/A               | 20331.1 (6668.5x)  |    N/A           |    N/A            |   68.5 (101.9x)   | 1.6.1        | julia     | 
| Julia     | Yes        |    N/A             |    N/A               | 16652.0 (5461.8x)  |    N/A           |    N/A            |   61.7 (91.9x)    | 1.6.1        | julia     |

So for the Run Time we get the result N/A.

If I run Julia from the command line and execute:

julia> @time main()
  0.000000 seconds
495802636

So the the execution time is zero, because the result is already calculated during compile time using constant propagation.

This means in the result table you should also report zero run time.

Or modify the test code so that full constant propagation is not possible.

nordlow commented 3 years ago

The constants in the generated code are randomized with a seed depending the current time and date to make caching impossible and comparison fairer so I see no reason to not closed this issue. See https://github.com/nordlow/compiler-benchmark#compiler-object-caches for details.

Currently I'm assuming that compilation is much slower than the run-time and bake these into the same metric for Julia. Would it be possible to alleviate this by writing a Julia wrapper script that first benchmarks the compilation and then the running of the generated main.jl enabling the Run-Time column to be populated by a real metric?

PallHaraldsson commented 1 year ago

It should be possible to use PackageCompiler.jl to compile the code, and then run it separately.