search

smarr / ReBenchDB

ReBenchDB records benchmark results and provides customizable reporting to track and analyze run-time performance of software programs.
MIT License
12 stars 6 forks source link

Comparison View: Indicates Missing Data, but data should be complete #172

Open smarr opened 8 months ago

smarr commented 8 months ago

These indicate missing data:

While the previous comparison on master looks complete: https://rebench.dev/TruffleSOM/compare/20f8545e990e4c82524885b8942f66bfea7d9b55..a33ffee806522eda1d9ab2405717fac3d1b91fb9#micro-steady-TruffleSOM-graal

The missing data is from yuria and yuria3, could theoretically be a race condition, but might also be another bug, for instance, identifying matching data correctly...

smarr commented 7 months ago

For the ReBenchDB project, there was a change in path for the Node.js binary since changing the underlying Docker image. However, there's only those things to compare directly, so, we should really be able to see the difference. In an ideal world, of course, we would know the change in software as well, and be able to display it, but that's not implemented.

https://rebench.dev/ReBenchDB/compare/1245e6bdd20bc0fe9b630fafe5e4ccd27512a8dd..bab07f534511c0a2009aadaca0573a8c3edfd819

(the situation here is also a little more complicated, because I manually changed the runIds for a subset of the data)