Collect all test results and handle cancelled tests properly

FoundationDB / fdb-joshua

FoundationDB Correctness service

Apache License 2.0

28 stars 25 forks source link

Open sfc-gh-kmakino opened 3 years ago

sfc-gh-kmakino commented 3 years ago

This PR addresses 3 issues:

When multiple tests start simultaneously, started can go beyond max_runs. In this scenario, the agent should wait for all tests to complete, rather than stop at max_runs and ignore the still running jobs
When agents die or considered to be dead due to not heart beating, we should ignore the cancelled tests as we don't know the results
When enough agents are running to serve available ensembles, other agents should timeout

ammolitor commented 3 years ago

Given that the unit tests failed (in the Github Actions build), I think this needs another look.

sfc-gh-kmakino commented 3 years ago

@ammolitor Can you tell what failed? build.sh works totally fine locally here.

sfc-gh-kmakino commented 3 years ago

@ammolitor CI passed. It would great if you can take another quick look. Thanks!

sfc-gh-kmakino commented 3 years ago

Now I realized the scaler needs to be aware of this change. Converting this as a draft for now.