search

niklas-heer / speed-comparison

A repo which compares the speed of different programming languages.
https://niklas-heer.github.io/speed-comparison
MIT License
475 stars 76 forks source link

Something is happening, and interfering, with testing #101

Open PallHaraldsson opened 1 year ago

PallHaraldsson commented 1 year ago

All numbers go up (for apparently no good reason, at least see no recent PRs explaining and such should affect individual languages): https://niklas-heer.github.io/speed-comparison/pages/2022-11-14T142222.html

e.g. Julia (AOT compiled) which was fastest, so not all go up by same fraction. But the machine is likely loaded while testing. It's still interesting that Julia goes up more than the others, or might well be a random factor I just notice more, since I'm interesting in Julia.

How do you test (and make sure not loaded, or should I say not make sure)? I usually use hyperfine myself. And if important to get right turn off (or disable) the web browser. For a testing machine, noting else should be running, not practical for myself.

This has happened before: https://niklas-heer.github.io/speed-comparison/pages/2022-10-24T132444.html

niklas-heer commented 1 year ago

@PallHaraldsson the tests are run through GitHub Actions on GitHubs/Azures infrastructure inside VMs. They get automatically started by earthly. The used infrastructures varies and that can contribute to difference in results. See #60.