Retry regressions in workers automatically

[Mark-Simulacrum]

This is easier to implement in workers rather than in the agent because we don't need to keep track of the run count or concatenate logs within the database. It does mean we get slightly weaker fault isolation between runs (e.g. if one machine is having a bad day or low on memory or whatever), but this is still an improvement.

This also means that the time taken per-regression increases, though an exact factor is hard to work out (we never retry baseline builds, and we only retry the first failed step...). Somewhere less than 5x though. In practice the expectation is that most of our runs have very few regressions (order of 1000 failed steps) compared to ~500,000 total steps executed. That is ~0.2%. So even a large increase here is likely to be acceptable.

If there's a large amount of regressions a single crater run may run really slowly -- but probably it is also not very useful as a run, so we may want some early-exit clause or confirmation step in that case anyway. For now not too worried about that case.

[Mark-Simulacrum]

@bors r+

[bors]

:pushpin: Commit dbdd723a6ece81548c3115fa5b15b1cf2f99e9fb has been approved by Mark-Simulacrum

It is now in the queue for this repository.

[bors]

:hourglass: Testing commit dbdd723a6ece81548c3115fa5b15b1cf2f99e9fb with merge b0abe1cb16c762a3721f334674e562c3d579731c...

[bors]

:sunny: Test successful - checks-actions Approved by: Mark-Simulacrum Pushing b0abe1cb16c762a3721f334674e562c3d579731c to master...