Add saving of logs to disk & combining into CSV

[michaeltremeer]

The code adds the following:

• Adds a json-save-dir arg to the load parser, so that JSON logs can be saved to disk

  • A filehandler is added to the base logger, and the output folders & file created automatically.
  • Pipes the logs out, including most of the run args (so that each log file contains all relevant configuration when being compared later).

• Adds a combine_logs subcommand to the parser

  • Extracts run information and final metrics (prior to termination/max requests/duration) from raw log files, and combines them into a single output CSV file, ready for analysis.

[michaeltremeer]

Good points in the comments, and the combine_logs functionality is a stretch for the core functionality of the repo - agreed. The only note I would say though (inspired by my own use): With shell redirection, you do miss any warnings or errors that occurred during the run, as well as any run parameters which make it easier to tell what the config was for each run (useful when you're analysing a number of permutations of shape, model and concurrency). You also miss the time the run finished or was terminated (which means the post-test period where requests are still draining can impact the numbers).

If you think it's valid and worth a change, I can resubmit just the first set of changes for saving a single run's logs, otherwise I can just submit the updated README for clarity.

[technicianted]

you do miss any warnings or errors that occurred during the run You can redirect both stdout and stderr to the same file if needed:

$ python -m benchmark.bench load ... > my/output/dir/full_output.log 2>&1

or separate files:

$ python -m benchmark.bench load ... > my/output/dir/stats_output.log 2> my/output/dir/logs.log

Does that cover the use-case?

as well as any run parameters which make it easier to tell what the config was for each run

Yes I agree this is super useful. Let's add that in a separate PR.

I can resubmit just the first set of changes for saving a single run's logs, otherwise I can just submit the updated README for clarity.

Yes let's break them down. Generally many smaller PRs is always better :)

Thanks!