ci: Docker In Docker sidecar dying will led to a failed build and not a canceled one.

[jhchabran]

This failed because of docker daemon going down:

• I redeployed agents
• at the same time was ran https://buildkite.com/sourcegraph/sourcegraph/builds/111980
• failed because the Docker Daemon was not responding
• I suspect that the DinD sidecar shut down before the agent.
  • edit: Reading about pod terminationmakes me think that's probably because the DinD shutdowns faster than the agent.

The big problem here is that the build is a failure, when in reality it is being canceled. This will mess with observability if unaddressed.

Big thanks to @mrnugget for taking the time to jump and pair on his failed build to have a closer look.

[jhchabran]

Made some tests, anything happening to an agent (killed, lost, whatever) is being displayed as a failure, unless ... you start looking at the exit code at the very bottom or in the timeline. In some cases, it is being displayed as a "cancellation", but not in the one below where I manually killed the pod.

Looks like a failure 👇

But in reality👇