search

bencheeorg / benchee

Easy and extensible benchmarking in Elixir providing you with lots of statistics!
MIT License
1.42k stars 66 forks source link

Benchmarking between BEAM versions #430

Closed RobinMorisset closed 6 months ago

RobinMorisset commented 6 months ago

Benchee seems great for comparing different functions on the same VM. Is there a way to use it for comparing the same function between two different VMs instead? It sounds like a lot of the functionality (e.g. warmup, statistical tests, plotting of the results, etc..) could be reused for this usecase. If it is not currently supported, would a PR be welcome?

RobinMorisset commented 6 months ago

Nevermind, I had missed the section on saving/loading results in the README, this is exactly what I needed.

PragTob commented 5 months ago

@RobinMorisset yes exactly! See: https://pragtob.wordpress.com/2024/01/08/tail-recursive-body-recursive-function-performance-across-elixir-beam-versions-whats-the-impact-of-the-jit/

I think I never ended up writing the separate blog post about how to do it, but here's the script that I used to do it: https://github.com/PragTob/tco_elixir/blob/main/run_bench.sh

Happy benchmarking!