Currently, the runner creates a complicated directory hierarchy to organize benchmarks. This both good and bad...
good: To send results for only one benchmark suite, all you have to do is send the one directory. (But then again, if you want to send results for the one benchmark suite but for all environments, then you have to send three directories and do some glob that's annoying to write)
bad: To aggregate results, we have to know about this directory structure in the aggregator
bad: To aggregate manually run results with existing aggregator configs, we have to create directories
We should rethink this and other parts of the runner-aggregator interface to simplify what's going on and make it easier to do things like manually run the aggregator on a new collection of files that aren't organized in the complicated way.
Of course, adding a command-line option to the aggregator telling it to load files from a particular directory would work to solve this issue... but how would it work when config files reference each other? Would paths mentioned in aggregator recipes be relative to the specified path instead?
There's probably more considerations for this issue...
What we need to pass from runner to aggregator
Actual data given by the benchmarks
Parameters used to launch the benchmarks. Can include command line
Environment and machine information
Possibly even the "raw" aggregator config used to read raw results into a pandas dataframe
Currently, the runner creates a complicated directory hierarchy to organize benchmarks. This both good and bad...
We should rethink this and other parts of the runner-aggregator interface to simplify what's going on and make it easier to do things like manually run the aggregator on a new collection of files that aren't organized in the complicated way.
Of course, adding a command-line option to the aggregator telling it to load files from a particular directory would work to solve this issue... but how would it work when config files reference each other? Would paths mentioned in aggregator recipes be relative to the specified path instead?
There's probably more considerations for this issue...
What we need to pass from runner to aggregator