search

MFlowCode / MFC

Exascale simulation of multiphase/physics fluid dynamics
https://mflowcode.github.io
MIT License
142 stars 65 forks source link

CI Benchmarking reports opposite results #395

Closed sbryngelson closed 4 months ago

sbryngelson commented 5 months ago

The CI Benchmarker reports master is X times faster than PR (or something like this) but in reality it is actually reporting master is X times slower than PR. This is a lhs vs rhs swap issue in bench.py.

sbryngelson commented 5 months ago

we were interpreting the results incorrectly, and the current message is technically correct but confusing. a more clear wording of this message would be helpful (or just an additional sentence like 1.5x indicates the PR is twice as fast as Master, 0.5x the PR is twice as slow. @anandrdbz

anandrdbz commented 5 months ago

Hmm I guess the problem is either way you're gonna run into an issue when x < 1, since it's much more natural to interpret values > 1. And the problem is this line doesn't actually use a value for x, it is simply written to indicate what the table values are.

sbryngelson commented 5 months ago

We could add a comment/line before the table that reads

For example: 1.5x indicates the PR is 1.5-times faster than Master; 0.5x indicates the PR is 0.5-times as fast as Master (so, slower)

anandrdbz commented 5 months ago

Okay I can do that, and maybe we should do it from the perspective of the PR anyways since master is the baseline (so PR is x times slower/faster than master instead of master is x times slower/faster than PR)

sbryngelson commented 5 months ago

That's fine as long as it's correct and is instructive