search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.84k stars 5.45k forks source link

error getting cached app managed resources: cache: key is missing #14506

Open tooptoop4 opened 1 year ago

tooptoop4 commented 1 year ago

Describe the bug

in the UI i click on a pod, then popup with pod details shows an error 'cache: key is missing' and it doesn't show the pod details. i found that if i click refresh button on the app it does not help, getting same error when clicking on pod. but if i click sync button on app then going to the pod works.

To Reproduce

been using argo for over year and first time this has happened....

Version 2.6.12

Logs

time="2023-07-13T22:27:53Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResourceTree grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" grpc.time_ms=376.227 span.kind=server system=grpc
time="2023-07-13T22:27:53Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" grpc.time_ms=216.462 span.kind=server system=grpc
time="2023-07-13T22:27:53Z" level=info msg="received unary call /application.ApplicationService/RevisionMetadata" grpc.method=RevisionMetadata grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" revision:\"myrevis\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" span.kind=server system=grpc
time="2023-07-13T22:27:53Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=RevisionMetadata grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" grpc.time_ms=119.744 span.kind=server system=grpc
time="2023-07-13T22:27:56Z" level=info msg="received unary call /application.ApplicationService/ManagedResources" grpc.method=ManagedResources grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="applicationName:\"myapp\" namespace:\"auth\" name:\"mypod\" kind:\"Pod\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:56Z" span.kind=server system=grpc
time="2023-07-13T22:27:56Z" level=info msg="Requested app 'myapp' refresh"
time="2023-07-13T22:27:57Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:57Z" span.kind=server system=grpc
time="2023-07-13T22:27:57Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:57Z" grpc.time_ms=43.44 span.kind=server system=grpc
time="2023-07-13T22:27:57Z" level=error msg="finished unary call with code Unknown" error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:56Z" grpc.time_ms=371.861 span.kind=server system=grpc
time="2023-07-13T22:27:57Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:57Z" span.kind=server system=grpc
time="2023-07-13T22:27:57Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:57Z" grpc.time_ms=69.178 span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="received unary call /application.ApplicationService/ManagedResources" grpc.method=ManagedResources grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="applicationName:\"myapp\" namespace:\"auth\" name:\"mypod\" kind:\"Pod\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="Requested app 'myapp' refresh"
time="2023-07-13T22:27:59Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" grpc.time_ms=9.584 span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=error msg="finished unary call with code Unknown" error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" grpc.time_ms=305.273 span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.claims="{\"at_hash\":\"redacthash\",\"aud\":\"argo-cd\",\"c_hash\":\"redactchash\",\"email\":\"redactemail\",\"email_verified\":true,\"exp\":1689373612,\"iat\":1689287212,\"iss\":\"https://redactdex\",\"name\":\"redactuser\",\"sub\":\"redactsub\"}" grpc.request.content="name:\"myapp\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" span.kind=server system=grpc
time="2023-07-13T22:27:59Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:59Z" grpc.time_ms=16.361 span.kind=server system=grpc
time="2023-07-13T22:28:05Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2023-07-13T22:27:53Z" grpc.time_ms=12350.736 span.kind=server system=grpc
wilkko commented 1 year ago

After upgrading from ArgoCD 2.7.14 to 2.3.8 (to Openshift GitOps operator 1.10) started getting this same error when clicking on certain pods in the UI or when running argocd app diff on command line:

$ argocd app diff autoscale  --local config/autoscale/itest                  
ERRO[0001] finished unary call with code Unknown  error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2023-10-03T12:06:14+03:00" grpc.time_ms=562.284 span.kind=server system=grpc
FATA[0001] rpc error: code = Unknown desc = error getting cached app managed resources: cache: key is missing

Redis pod has similar looking key for it:

$ redis-cli --scan --pattern 'app|managed-resources|autoscale*'
app|managed-resources|autoscale|1.8.3.gz
wilkko commented 1 year ago

Using argocd login <server> --sso makes the ArgoCD CLI working again.

Issue can be reproduced with:

- oc login... (updates kubekonfig)
- argocd login --core
jblsk commented 8 months ago

We see a variant of this issue with ArgoCD v2.10.1 in argocd-application-controller log with recurring errors like: time="2024-03-05T14:00:11Z" level=error msg="DiffFromCache error: error getting managed resources for app <app_name>: cache: key is missing"

0xDones commented 7 months ago

I eventually get some of those errors on the logs, for every application, even when no one is using the UI or CLI. I wonder if in my case those are really errors or if they should be treated differently, because it may be triggering some false positive alerts.

time="2024-03-19T16:47:00Z" level=error msg="DiffFromCache error: error getting managed resources for app karpenter-dev: cache: key is missing"

ArgoCD version: v2.10.1

shm-eboks commented 3 months ago

seeing these errors pop up as well now

DiffFromCache error: error getting managed resources for app <xxx>: cache: key is missing

running argocd v2.11.4+e1284e1

omfurman-ma commented 3 months ago

We also experience this using:

argocd login --core --sso
 - > Context 'kubernetes' updated

argocd app diff .....
-> ERRO[0007] finished unary call with code Unknown         error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService  span.kind=server system=grpc
FATA[0007] rpc error: code = Unknown desc = error getting cached app managed resources: cache: key is missing
AllanMNoguera commented 3 months ago

argocd app sync works good only argocd app diff is having this issue when using --core

time="2024-07-23T15:19:18Z" level=error msg="finished unary call with code Unknown" error="error getting cached app managed resources: cache: key is missing" grpc.code=Unknown grpc.method=ManagedResources grpc.service=application.ApplicationService grpc.start_time="2024-07-23T15:19:14Z" grpc.time_ms=3983.76 span.kind=server system=grpc
time="2024-07-23T15:19:18Z" level=fatal msg="rpc error: code = Unknown desc = error getting cached app managed resources: cache: key is missing"