search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.65k stars 5.38k forks source link

Inconsistency in redis key formation in ArgoCD 2.8.4 with compression enabled #15912

Open purajit opened 1 year ago

purajit commented 1 year ago

Checklist:

Describe the bug

ArgoCD 2.8 enables Redis compression by default. However, after upgrading, I found I was unable to run commands like argocd app diff/manifest, since it would fail with 

ERRO[0031] finished unary call with code Unknown         error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-10-11T10:30:18-07:00" grpc.time_ms=1554.185 span.kind=server system=grpc

Everything else runs without issue - syncing, UI browsing, etc.

To reproduce Unsure what to mention here.

What I've tried that didn't help:

What works:

Current configuration:

Expected behavior

Everything should "just work", and key formation should be consistent.

Version

We're using the ArgoCD helm chart (v5.46.7).

# on server pod
> /usr/local/bin/argocd-server version
argocd-server: v2.8.4+c279299
  BuildDate: 2023-09-13T19:12:09Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64

# on local
> argocd version
argocd: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: darwin/arm64
  Kustomize Version: could not get kustomize version: exec: "kustomize": executable file not found in $PATH
  Helm Version: v3.9.0+g7ceeda6
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.20.0

Logs

I cleared out some cache keys, and then ran redis-cli monitor | grep <application> on the Redis pods:

1697046390.058300 [0 172.20.65.196:6379] "SET" "app|managed-resources|<application>|1.8.3.gz"
1697046390.060277 [0 172.20.65.196:6379] "SET" "app|managed-resources|<application>|1.8.3.gz"
1697046400.521017 [0 10.50.80.110:50238] "get" "app|managed-resources|<application>|1.8.3"

Note the lack of the suffix on the last one.

I could potentially add a ton of logging, build argo-cd and create an image, and deploy that to get more information, but I was hoping to see if anyone has dealt with this or knows what might be happening first. Looking at the code, I can't see why the key formation would ever be inconsistent.

Some relevant code paths:

crenshaw-dev commented 1 year ago

Are you passing any flags on the argocd app diff/manifests command, or just the app name?

ashinsabu3 commented 11 months ago

I have seen the same when I upgraded from argo 2.7.2 to argo 2.8.2, setting the redis.compression config to none seems to fix this.

cmontemuino commented 10 months ago

Also confirmed that adding redis.compression: none in ConfigMap argocd-cmd-params-cm solves the issue. I'm also using ArgoCD v2.8.4.

purajit commented 10 months ago

Are you passing any flags on the argocd app diff/manifests command, or just the app name?

Hey apologies, really need to figure out my GH notifications game.

Not passing in any other flags to the argocd commands. I don't have a cluster without compression available right now, but I believe I had this issue with and without --local as well.

tooptoop4 commented 5 months ago

same issue in 2.10.3. redis.compression: none as workaround solves

tooptoop4 commented 4 months ago

any idea? @aborilov

agaudreault commented 3 months ago

Might be related to https://github.com/argoproj/argo-cd/issues/5068