Allow comparison of 2 different executions of gradle profiler

[yogurtearl]

gradle-profiler --compare yesterday_benchmark.csv today_benchmark.csv

For any scenarios with the same name in the 2 files, gradle-profiler would output a compare.csv and compare.html that highlight the differences between the 2 runs.

This is useful comparing nightly benchmarks.

Also, would be useful to have an exit code that indicated performance regression of today_benchmark.csv compared to yesterday_benchmark.csv. Then we can produce an alert if there is a build performance regression.

[asodja]

That would be a nice feature indeed. I doubt that we will work on it now since there are other priorities in gradle/gradle atm. But you can provide a PR if you are interested.

I guess you could bypass all the scenario running logic and go directly to the report generation. I suggest that implementation points the --compare argument to the different profile-out locations and not directly to a csv, so we could generate differential flame graphs later on if we want to.

For easier comparison, you might want to work on the json, since csv doesn't contain all the information. I suggest you output the json that is embedded into a html as a separate file. So you would have benchmark.csv, benchmark.html and benchmark.json in profile-out folder. And then you can do a comparison on the json.