sharkdp / hyperfine

A command-line benchmarking tool
Apache License 2.0
20.69k stars 333 forks source link

Incorrect result for fastfetch #729

Closed FelixFourcolor closed 3 months ago

FelixFourcolor commented 3 months ago

Typical run time of fastfetch on my system:

$ time fastfetch >/dev/null
________________________________________________________
Executed in   33.64 millis    fish           external
   usr time   13.04 millis    0.00 micros   13.04 millis
   sys time   12.51 millis  622.00 micros   11.88 millis

Hyperfine's report:

$ hyperfine "fastfetch >/dev/null"

Benchmark 1: fastfetch >/dev/null
  Time (mean ± σ):      18.1 ms ±   3.2 ms    [User: 9.0 ms, System: 10.2 ms]
  Range (min … max):     7.2 ms …  31.3 ms    83 runs

Warning: The first benchmarking run for this command was significantly slower than the rest (31.3 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Only the first run accurately represents fastfetch's typical run time, subsequent runs are much faster. Running hyperfine with --warmup like the warning suggests does not change the result.

(Arch linux, hyperfine 1.18.0, fastfetch 2.8.8, both installed from arch official repo.)

sharkdp commented 3 months ago

This seems completely normal. The first run is slow, while subsequent runs are faster. Presumably due to disk caches. Try running time fastfetch in fast succession and you should see similar results.

And please read this section: https://github.com/sharkdp/hyperfine?tab=readme-ov-file#warmup-runs-and-preparation-commands which explains how to do warm-cache and/or cold-cache benchmarks.

FelixFourcolor commented 3 months ago

Try running time fastfetch in fast succession and you should see similar results.

You're right. My bad, I should've tried that before opening an issue.