smarr / ReBenchDB

ReBenchDB records benchmark results and provides customizable reporting to track and analyze run-time performance of software programs.
MIT License
12 stars 6 forks source link

Compare View, Executor Comparison: Handling of 0 Results #203

Open smarr opened 3 months ago

smarr commented 3 months ago

This example shows 0 results for GraphSearch on SOM-RS: https://rebench.stefan-marr.de/som-rs/compare/61927bdbe704dc514d3c2744f1351cfe483733d7..0831a9251692c20675e347f860099995b96a4850#exe-comparisons

I would guess the data is missing here, and should not be included. If ReBench however actually recorded a 0 and reported it as such, I am not sure what the most robust approach to handle it is. For some metrics, 0 is a perfectly valid value. Though, for time, we would assume it indicates a problem.