argocd app logs cli doesn't gracefally report deleted logs

[KrE80r]

• Version:
  • Server: v2.3.6+0badce7 (probably latest version affected too)
  • cli : argocd: v2.5.4+86b2dde

Issue description:

argocd "app logs" cli throws a vague error

# argocd app logs --auth-token ... \                                                                                                                                                                                                              
      --server myserver.com app-name

FATA[0000] stream read failed: rpc error: code = Unknown desc = parsing time "unable" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "unable" as "2006"

After some investigation it seems this is caused by a one container within the pod with no logs on disk

# kubectl logs   pods/app-name-restart-1234567-xyz
unable to retrieve container logs for docker://7e18e10a858d485aec93792c3f60ad3aa133a15cc230dd2d70095d9aeb3d2684

The "unable" above is what triggers the "cannot parse" error since argocd expects a timestamp at the beginning instead.

Expected behavior

Argocd to report on missing container logs and printout other containers (within the pod) logs instead of failing.

[asfaltboy]

I've also been impacted by this issue, below is the recounting of my experience.

We have been running ArgoCD on GCP for a few years, and use it to manage apps deployed to GKE clusters. We often see references to older pods in ArgoCD, until they get garbage collected, which means it impacts many of our apps, until we manually clear older replica sets or pods.

For illustration, the user journey impact can be described ass such:

1. Open an app with some stopped pods.
2. Click a Depoyment and then click the logs tab
3. The logs fail to load and the message "Failed to load data, please try again" is shown in the content
4. A toast notification shows up with the error:
5. The server logs show the same cannot parse error as in the issue above

Note: I have seen this behavior ever since I first used Argo (version 2.2.5 through 2.6.7). I am a bit surprised that no one else reported this before, or at least I could not find issues. I wonder if other users of ArgoCD have a quick fix for this, or another workaround (that isn't in FAQ yet) 🤔

[artem-zherdiev-ingio]

+1

[jsurana]

I am also getting error while terminating POD rpc error: code = Unknown desc = parsing time "unable" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "unable" as "2006". PODs are not getting terminated successfully. It's showing terminating only. Do we have any fix for this? FYI, I don't see any reason in application to throw such kind of error