[SYNTH-16456] Postpone reporting results on 404

[Drarig29]

What and why?

There is sometimes some latency between batches and the poll results endpoint. When this happens, the poll results endpoint may return a 404 error because some queried results cannot be found, so the request ends up returning nothing.

Currently, we retry 3 times on 404 errors for that endpoint, and if the request never succeeds, datadog-ci quits and the CI fails.

How?

• Extend our concept of incomplete results to also take care of results that have a 404: the reporting is postponed, then the incomplete results are fetched in next polling cycles.
• We backup polling results in case the last polling cycle encounters a 404, because this last polling cycle has to fetch all results and we'll probably have fetched and reported most results already.
• Like before, incomplete results may still be incomplete at the end of the batch: in that case, the result is reported without a server result and the following log is printed: The information for result {resultId} of test {publicId} was incomplete at the end of the batch.

Review checklist

• [x] Feature or bugfix MUST have appropriate tests (unit, integration)

[datadog-datadog-prod-us1[bot]]

Datadog Report

Branch report: corentin.girard/SYNTH-16456/incomplete-results-when-poll-result-404 Commit report: a91b3fa Test service: datadog-ci-tests

:white_check_mark: 0 Failed, 1640 Passed, 0 Skipped, 1m 57.02s Total duration (1m 33.05s time saved)