Bodigrim / tasty-bench

Featherlight benchmark framework, drop-in replacement for criterion and gauge.
https://hackage.haskell.org/package/tasty-bench
MIT License
80 stars 11 forks source link

Do not overwrite CSV file, instead append to it #17

Closed harendra-kumar closed 3 years ago

harendra-kumar commented 3 years ago

tasty-bench overwrites the CSV file every time, whereas the behavior of gauge is to append to it. The latter is more convenient as we can keep appending measurements to the same CSV file and then the tools processing the CSV file can select or compare the measurements.

With the appending behavior as default it is easy to get the overwriting behavior by just removing the existing file before performing measurements. Alternatively, a CLI option can be provided to select the appending vs overwrite behavior. Though its always better to have fewer CLI options so I would prefer the appending behavior by default and no CLI option.

Bodigrim commented 3 years ago

gauge does not need to parse its CSV output, so it is easy to append them without much care. But tasty-bench with --baseline reads results from a previous run to report relative comparisons. It would be impossible to do so, if the baseline contains several runs appended.

I can assign a special meaning to --csv -, dumping csv to stdout, so that a caller is able to redirect it whenever desirable. (Other output can be suppressed by --quiet)

Bodigrim commented 3 years ago

On the second thought, dumping CSV to stdout would interfere badly with --hide-successes and probably does not buy us much: one can dump CSV to a temporary file and append it to the main one later.

harendra-kumar commented 3 years ago

I have dealt with it in the measuring script for now, by writing to a temp file and then appending the results to another file. If it poses other problems, we can close the issue and move on.