ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
326 stars 25 forks source link

`benchmark check` to report every comparison even if successful #270

Open MahdiBM opened 2 months ago

MahdiBM commented 2 months ago

AFAIK benchmark check only report the failed checks. I want it to always report everything. Even if nothing has failed. Perhaps the failed checks should be at the beginning of the report, and then the checks that passed.

Currently we use benchmark read to read the results and attach them to a PR comment, but that doesn't contain the comparisons at all, so while the checks have passed, I can't immediately and easily compare the passed checks to the static thresholds that we're using.

hassila commented 1 month ago

It’s very intentional to not log everything - some benchmarks suites are quite large and it becomes very verbose -,the idea is to focus on deviations - maybe a verbose flag could be considered for that if it’s desired?

MahdiBM commented 1 month ago

Ah right, I didn't mean that the default should necessarily change. An option would be enough.