argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
18.05k stars 5.51k forks source link

Dangling `.git/HEAD.lock` causes flakiness in e2e tests #21017

Open andrii-korotkov-verkada opened 2 hours ago

andrii-korotkov-verkada commented 2 hours ago

Checklist:

Describe the bug

The issue is that either git crashes or there's a concurrency issue, leading to the mentioned file existing and failing the test run. Example: https://github.com/argoproj/argo-cd/actions/runs/12109162717/job/33758025750?pr=20998

rpc error: code = Internal desc = Failed to checkout FETCH_HEAD: `git checkout --force FETCH_HEAD` failed exit status 128: fatal: update_ref failed for ref 'HEAD': cannot lock ref 'HEAD': Unable to create '<path to cached source>/.git/HEAD.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n

To Reproduce

It just happens from time to time.

Expected behavior

Dangling lock file doesn't fail the tests.

Screenshots

Screenshot 2024-12-01 at 3 42 28 PM

Version

Logs

andrii-korotkov-verkada commented 2 hours ago

I wonder if the file should be cleaned up as a part of EnsureCleanState if it's not already, or maybe before running each git command.