RFC: Split the benchmarks into per package files?

[harendra-kumar]

Currently the benchmarks for all packages are in a single file.

What are the advantages of this:

• The biggest reason is that the benchmarks being compared are run in sequence one after another. For example, the fold operations for each package run close together one after another. The benefit of this is that the comparison is more accurate as the macro situation e.g. the cpu load etc. does not change much in that period of time. If we ran all the benchmarks for one package and then all the benchmarks for another then when we compare individual operations they may vary according to the load at the time since they both ran far apart in time.

What are the disadvantages:

• Dependency issues make it harder to keep it all lumped together and working. If we have separate files we can even use separate cabal files to build them independently. The code will perhaps be more modular in separate files. And we can almost use the same code in each file, just change the imports and rename some ops.

[harendra-kumar]

If we use per package files we can use a separate driver process to run the benchmarks close together in time. That's just more work to do. But it may make sense to create a general benchmark driver package to perform such tasks.

[harendra-kumar]

I split the definition of benchmarking ops of each package to individual files though we still have a monolithic high level driver. This makes understanding existing benchmarks and adding new benchmarks easier.