Additional Result Comparison request

faster-cpython / bench_runner

Code for running pyperformance benchmarks on Github Action runners

BSD 3-Clause "New" or "Revised" License

4 stars 5 forks source link

Additional Result Comparison request #276

Closed yuqiw4 closed 1 month ago

yuqiw4 commented 1 month ago

Now when we started two workflows, one with no flag, another with NOGIL enabled, then the bench_runner will output:

Merge base (no flag) vs PR/ref (no flag)
Merge base (NOGIL) vs PR/ref (NOGIL)

Can we also add an additional comparison:

Merge base (no flag) vs PR/ref (NOGIL)

yuqiw4 commented 1 month ago

I've done some investigation and think we might be able to achieve it by revising some methods in result.py.

cc @mdboom , please let me know your thought. Thank you soooo much!

mdboom commented 1 month ago

Yep, that seems like a good idea, but maybe we should make it a configuration option. We don't want to do all possible comparisons for space reasons. We didn't think this was an important one for our team's results, but I can completely understand how it would be for someone else's.