sharkdp / hyperfine

A command-line benchmarking tool
Apache License 2.0
22.04k stars 358 forks source link

Add an option to save a benchmark run for later comparisons #607

Open CosmicHorrorDev opened 1 year ago

CosmicHorrorDev commented 1 year ago

There are times when I want to have some baseline saved that I compare later changes to. It would be convenient if I had a way to save a benchmarking run and then load that to later runs. Something like

$ hyperfine --save-benchmark baseline 'sleep 1'
$ hyperfine --load-benchmark baseline 'sleep 0.5'
{{ display things as though `hyperfine 'sleep 1' 'sleep 0.5'` had been run }}

If you're interested in supporting this then I can work on adding it as a feature

sharkdp commented 1 year ago

That sounds like a cool idea, thank you. It also sounds like a rather complex feature that is not easy to design. So before we go ahead trying to implement this, I'd like to discuss how this would work, what kind of implications this has on other features, what the CLI would look like, etc.

Note: we already have --export-json, so that could probably be used for the "storing" side of things.

sharkdp commented 1 year ago

Slightly related: #577

miluoshi commented 6 days ago

I would also welcome this feature. It's currently possible to save report with --export-json so could it be implemented in a way, that instead of defining one command, there would be a --import-json flag?

# before - a single run
hyperfine 'command1' 'command2'

# after - two separate runs
hyperfine 'command1' --export-json baseline.json
hyperfine --import-json baseline.json 'command2'

The position of arguments would determine the order of compared commands the same way as position of commands in the 1st example. What do you think?