Run for more rounds?

[mcabbott]

Running this longer gives about 3 more digits of accuracy:

julia> e3 = f(10^3) - pi
-0.0009980037435091127

julia> e6 = f(10^6) - pi
-9.999980186137236e-7

julia> e9 = f(10^9) - pi
-1.0017426887998226e-9

julia> e3 / e6
998.0057209439519

julia> e6 / e9
998.2583649418102

It also changes it from a measure of startup time to a measure of computation time. This certainly matters for Julia, #22, and perhaps also for Java, LuaJit, PyPy?

Edit: second commit has this CI run https://github.com/niklas-heer/speed-comparison/actions/runs/3257449769 which took 50 mins, and produced this:

That's rounds = 10^9. See here below for rounds = 10^8.

[niklas-heer]

Mhm... the CI has been running for ~20 minutes. This is definitely too much. Maybe we could then also reduce the number of times it executes the programs from 10 to 3 to determine the best average?

[mcabbott]

Or set it to 1 for slow languages?

[Moelf]

This is definitely too much.

well, so each time can be separated into two parts:

• fixed overhead
• (time for 1 round) * nrounds

for Julia, the "fixed overhead" is like 80ms, and the "time for 1 round" is very small.

For Python or Clojure, the "time for 1 round" is much larger, so when you multiply 1000x, they become visible

[giordano]

@niklas-heer I'm looking at the CI log of the first and second commit of this PR. The only difference, introduced in the second commit, is the number of iterations, but with 10 iterations I see

                +cpp | Benchmarking Summary 
                +cpp | -------------------- 
                +cpp | Started:            15-10-2022 23:00:30 
                +cpp | Ended:              15-10-2022 23:00:45 
                +cpp | Language:           C++ (g++) 
                +cpp | Version:            11.2.1 
                +cpp | Executed Command:   ./leibniz 
                +cpp | Accuracy:           0.6666666666666666 
                +cpp | Calculated Pi:      3.1415926545880506 
                +cpp | Total iterations:   10 
                +cpp | Average time taken: 1.532409457s 

while with 3 iterations

                +cpp | Benchmarking Summary 
                +cpp | -------------------- 
                +cpp | Started:            15-10-2022 23:17:40 
                +cpp | Ended:              15-10-2022 23:17:54 
                +cpp | Language:           C++ (g++) 
                +cpp | Version:            11.2.1 
                +cpp | Executed Command:   ./leibniz 
                +cpp | Accuracy:           0.6666666666666666 
                +cpp | Calculated Pi:      3.1415926545880506 
                +cpp | Total iterations:   3 
                +cpp | Average time taken: 4.909791779s 

I was expecting "Average time taken" to be roughly the same, or am I getting wrong what that means?

[Moelf]

I think there's one more place to change 10 -> 3, this time difference looks like *10 / 3 kind of thing.

[niklas-heer]

No, there is only one place to change the number of iterations. The GitHub Action run is executed on a VM. I don't know if parallel CI runs affect each other, but even with a reduction to 3 iterations, it is now running over 40 minutes. That is still too much. I think we have to find a lower rounds value which doesn't exceed 10 minutes each CI run.

original:      1000000
proposed:  1000000000

Let's try 10000000 so one 0 more compared to the original times 3 iterations.

[giordano]

No, there is only one place to change the number of iterations.

Don't you agree that the average should be roughly the same if using 3 or 10 iterations (especially for programs with stable runtime like C/C++)? How do you explain the difference I reported above?

[Moelf]

https://github.com/niklas-heer/speed-comparison/blob/29a93c95646731c46c42e5739ee3bffabb7c83d4/scbench/cmd/scbench/main.go#L122

this ten seems hard-coded?

[Moelf]

https://github.com/niklas-heer/speed-comparison/blob/master/src/leibniz.swift#L1

swift has the hard-coded old number;

and yeah looks like we can remove Julia + PackageCompiler

[niklas-heer]

https://github.com/niklas-heer/speed-comparison/blob/29a93c95646731c46c42e5739ee3bffabb7c83d4/scbench/cmd/scbench/main.go#L122

this ten seems hard-coded?

Yes, this is the default value. If you don't provide anything there, it takes 10.

[niklas-heer]

It runs 10 minutes, so I wouldn't increase the rounds.txt any more. The results have changed quite a bit.

This is the results: (I guess we should ignore Swift)

[Moelf]

yeah and let's remove PackageCompiler, which should save 2 minutes I guess?

but honestly, knowing 50% time in Julia is still start-up time makes me sad

[mcabbott]

Something is broken with the time calculations. C is 4985ms with 10^9 rounds above, and 162ms with 10^8. The ratio should be close to 10, surely?

[jariji]

I'd keep PackageCompiler.jl, at least until StaticCompiler.jl is ready - it's interesting to see the comparison.

[niklas-heer]

@Moelf I'll merge it as it is an will keeped the compiled version like @jariji suggested. 10 minutes is okay.

[niklas-heer]

Something is broken with the time calculations. C is 4985ms with 10^9 rounds above, and 162ms with 10^8. The ratio should be close to 10, surely?

I wouldn't give too many thoughts about a single run from a CI build which ran parallel with others and which I had to cancel in the end because it got stuck.

[giordano]

I wouldn't give too many thoughts about a single run

Single run? We reported two different runs which had inconsistent results.

from a CI build which ran parallel with others

In parallel with what? Each job runs in a different virtual machine.

and which I had to cancel in the end because it got stuck.

It wasn't stuck, it was running CPython