search

smarr / ReBenchDB

ReBenchDB records benchmark results and provides customizable reporting to track and analyze run-time performance of software programs.
MIT License
13 stars 6 forks source link

Possible Race between last Result and Report Generation #33

Open smarr opened 4 years ago

smarr commented 4 years ago

There is a possible race between the last result being submitted, getting the OK from ReBenchDB, and processing the completion from ReBench.

The race is possible because the data input is done after returning the OK back from ReBenchDB, and the completion signal can be theoretically be processed before the data is completely in the database.

Example job:

https://git.cs.kent.ac.uk/sm951/RPySOM/-/jobs/13390

The last benchmark PageRank, is marked as missing in the report:

https://rebench.stefan-marr.de/compare/RPySOM/f81659fa88083b49b917fa47b8091cfc1fe1449a/357cc88a43e04a73920a4b82b7ee3d203d17f9c6

smarr commented 4 years ago

The report to the previous version, instead of the main branch, shows the data to be in the database. So, that pretty much confirms the race.

https://rebench.stefan-marr.de/compare/SOMns/51856e23461afbd1e51c57a92e97c7c6f6f10da2/357cc88a43e04a73920a4b82b7ee3d203d17f9c6