buildkite-agent 3.0 removes checkout on failure

[joscha]

With the old 2.x agent, a partial checkout on a failed build was kept. Due to a big git LFS checkout, builds with small timeouts (2min) that can't check out the repository completely any more now continuously fail. With the 2.x agent, they would eventually (after a few re-runs) acquire a full checkout. I looked at the docs but can't see any option that would allow us to keep this behaviour. Alternatively #347 would work.

cc @arromer

[lox]

That's a tricky one! We clear the checkout because there are several ways that partial checkouts can end up corrupt and cleaning them on failure prevents failure loops. I guess the obvious question is why the 2 minute timeout?

The other thing we'd like to do eventually is move to git work trees for checkouts along the lines of https://github.com/sj26/git-worktree-buildkite-hooks. I wonder if that approach might work for your project?

[joscha]

we have the short timeout because sometimes the build step stalls and we don't want to have a 10-minute stalled build of something that should take no more than 30 seconds just because our checkout takes a while. I think either dynamic checkouts or keeping the checkout with a separate timeout would be best. git-worktree would probably work as well, however even with worktree, there needs to be an initial checkout per agent.