Closed harendra-kumar closed 3 years ago
gauge
does not need to parse its CSV output, so it is easy to append them without much care. But tasty-bench
with --baseline
reads results from a previous run to report relative comparisons. It would be impossible to do so, if the baseline contains several runs appended.
I can assign a special meaning to --csv -
, dumping csv to stdout, so that a caller is able to redirect it whenever desirable. (Other output can be suppressed by --quiet
)
On the second thought, dumping CSV to stdout would interfere badly with --hide-successes
and probably does not buy us much: one can dump CSV to a temporary file and append it to the main one later.
I have dealt with it in the measuring script for now, by writing to a temp file and then appending the results to another file. If it poses other problems, we can close the issue and move on.
tasty-bench
overwrites the CSV file every time, whereas the behavior ofgauge
is to append to it. The latter is more convenient as we can keep appending measurements to the same CSV file and then the tools processing the CSV file can select or compare the measurements.With the appending behavior as default it is easy to get the overwriting behavior by just removing the existing file before performing measurements. Alternatively, a CLI option can be provided to select the appending vs overwrite behavior. Though its always better to have fewer CLI options so I would prefer the appending behavior by default and no CLI option.