Redis issue on argocd-application-controller v2.13.0, chart 7.7.3

[davidt-gh]

Checklist:

• [ V ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ V ] I've included steps to reproduce the bug.
• [ V ] I've pasted the output of argocd version.

Describe the bug The controller was unhealthy (the UI was stuck). Then looked the logs, saw some info logs, and after moment got a lot of logs (which in case I was able to catch) Started with:

Reconnect to redis because error: \"dial tcp: lookup argocd-redis: i/o timeout\

1 second later the whole log (Which attached in here), but starting with:

fatal error: sync: unlock of unlocked mutex

To Reproduce Changed argocd helm chart 7.6.12 -> 7.7.3

Expected behavior Have small downtime, and keep working :)

Screenshots

Version

argocd: v2.12.6+4dab5bd
  BuildDate: 2024-10-18T21:32:12Z
  GitCommit: 4dab5bd6a60adea12e084ad23519e35b710060a2
  GitTreeState: clean
  GoVersion: go1.23.2
  Compiler: gc
  Platform: darwin/arm64
WARN[0001] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
argocd-server: v2.13.0+347f221

Logs

Paste any relevant application logs here.

application-controller.log

[andrii-korotkov-verkada]

Looks like an issue with Redis library. They did some changes on master for the relevant code https://github.com/redis/go-redis/blob/master/redis.go#L178-L180, https://github.com/redis/go-redis/commit/080e051124d5e35a0d0c49e03f8547e86365c5d8. Let's wait for a release of 9.7.1 or 9.8.0 and upgrade.

[andrii-korotkov-verkada]

I still don't understand how it becomes unlocked.

[andrii-korotkov-verkada]

Can you file an issue for Redis and share a part of logs and this issue, please? This may help to get cherry pick into 9.7.1 earlier.

[davidt-gh]

Can you file an issue for Redis and share a part of logs and this issue, please? This may help to get cherry pick into 9.7.1 earlier.

https://github.com/redis/go-redis/issues/3195 - please add comment or context if needed.