search

sharkdp / hyperfine

A command-line benchmarking tool
Apache License 2.0
22.24k stars 358 forks source link

Exporting detailed user/system times (in the JSON output)? #660

Open gendx opened 1 year ago

gendx commented 1 year ago

Currently, the wall time is reported with detailed statistics both in the TUI (mean + stddev + median + min + max), and in the JSON output (list of all the wall times via the times field).

However, the user and system times only report one number. The UI doesn't specify which number it is, but looking at the code reveals that it's the mean (https://github.com/sharkdp/hyperfine/blob/master/src/benchmark/mod.rs#L376-L377).

In general, I think that the other statistics are relevant, as the mean can suffer from outlier bias and doesn't show the whole picture. And of course a detailed distribution plot can be worthwhile in some cases (as the various scripts allow), which requires the list of all the recorded times.

As such, would you be open to export more details of the user/system times?

  1. At the minimum, I think that adding user_times and system_times arrays to the JSON output would allow further tools to reconstruct the median and stddev (or any other interesting statistic).
  2. Additionally, the JSON output could also be extended with user_stddev, user_median, user_min, user_max fields (and likewise for system).
  3. Lastly (but it's maybe more disruptive), the TUI could also report these details for user/system, when some CLI flag is enabled.

What do you think? I'll be happy to contribute PR(s) if you think some of these are worth adding.

sharkdp commented 1 year ago

However, the user and system times only report one number. The UI doesn't specify which number it is, but looking at the code reveals that it's the mean (https://github.com/sharkdp/hyperfine/blob/master/src/benchmark/mod.rs#L376-L377).

Yes. This should be better documented.

Slightly related: I also wanted to document what exactly user and system time mean, as that can be quite confusing (https://github.com/sharkdp/hyperfine/issues/597).

In general, I think that the other statistics are relevant, as the mean can suffer from outlier bias and doesn't show the whole picture. And of course a detailed distribution plot can be worthwhile in some cases (as the various scripts allow), which requires the list of all the recorded times.

Absolutely.

At the minimum, I think that adding user_times and system_times arrays to the JSON output would allow further tools to reconstruct the median and stddev (or any other interesting statistic).

Yes, let's do it.

Additionally, the JSON output could also be extended with user_stddev, user_median, user_min, user_max fields (and likewise for system).

Sounds good. It would be great if we could group those somehow. Something like:

Unfortunately, that would be a breaking change to the existing JSON format. But maybe something we could address in a 2.0 release?

Lastly (but it's maybe more disruptive), the TUI could also report these details for user/system, when some CLI flag is enabled.

That would be an option, but I'd like to carefully plan this feature (and the corresponding CLI). In the past, we also wanted to add the median run time to the output (https://github.com/sharkdp/hyperfine/pull/344), so the biggest question is: how should the new (default and possibly extended) layout look like?

What do you think? I'll be happy to contribute PR(s) if you think some of these are worth adding.

Contributions are always very welcome! Thank you.