"argocd app wait" hangs when the auth token expires during its execution

[yannicknghiemxuan]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

"argocd app wait" hangs when the auth token expires during its execution.

For example if you have a rollout in progress that takes 10 minutes to complete and your token is valid for 1 minute, then the token will expire during the "argocd app wait" execution and the command will hang forever (never returns).

To Reproduce

Example to reproduce the issue:

token=$(curl -s --insecure ${serverurl}/api/v1/session -d '{"username":"admin", "password":"xxxx"}' | jq .token -r)
authtoken=$(curl -X POST -H "Authorization: Bearer ${token}" -s --insecure ${serverurl}/api/v1/projects/boxever-services/roles/jenkins-remote-access/token -d '{"description": "Tmp jenkins token for build", "project": "boxever-services", "role": "jenkins-remote-access", "expiresIn": 60}' | jq .token -r)
argocd app wait myservices --grpc-web  --insecure --server ${serverurl}  --auth-token ${authtoken}

Expected behavior

If the token expires while the command is running, then it should interrupt its execution and return an error.

Screenshots

Version

ArgoCD cli Versions impacted: at least 2.3.4 and 2.3.3 We never hit the issue with 2.1.3, and we haven't tried the versions in between.

$ argocd version
argocd: v2.3.4+ac8b7df.dirty
  BuildDate: 2022-05-22T11:56:25Z
  GitCommit: ac8b7df9467ffcc0920b826c62c4b603a7bfed24
  GitTreeState: dirty
  GoVersion: go1.18.2
  Compiler: gc
  Platform: darwin/amd64

Logs

$ argocd app wait boxever-eks-multi-armed-bandit --grpc-web --insecure --server localhost:8886 --auth-token eyJhb...
TIMESTAMP                  GROUP                            KIND                NAMESPACE                              NAME                                 STATUS   HEALTH            HOOK  MESSAGE
2022-05-27T11:54:16+01:00                              ConfigMap               boxever-public-service  boxever-eks-multi-armed-bandit-configs-ffgkk7km5b    Synced                           configmap/boxever-eks-multi-armed-bandit-configs-ffgkk7km5b unchanged
2022-05-27T11:54:16+01:00  argoproj.io                   Rollout               boxever-public-service  boxever-eks-multi-armed-bandit                       Synced  Progressing              rollout.argoproj.io/boxever-eks-multi-armed-bandit configured
2022-05-27T11:54:16+01:00  argoproj.io                AnalysisTemplate         boxever-public-service  multi-armed-bandit-query-status-job-executing        Synced                           analysistemplate.argoproj.io/multi-armed-bandit-query-status-job-executing unchanged
2022-05-27T11:54:16+01:00                              ConfigMap               boxever-public-service  telegraf-boxever-eks-multi-armed-bandit              Synced                           configmap/telegraf-boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00  rbac.authorization.k8s.io  RoleBinding              boxever-public-service  boxever-eks-multi-armed-bandit                       Synced                           rolebinding.rbac.authorization.k8s.io/boxever-eks-multi-armed-bandit reconciled. rolebinding.rbac.authorization.k8s.io/boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00  autoscaling                HorizontalPodAutoscaler  boxever-public-service  boxever-eks-multi-armed-bandit                       Synced  Healthy                  horizontalpodautoscaler.autoscaling/boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00  monitoring.coreos.com      PrometheusRule           boxever-public-service  boxever-eks-multi-armed-bandit                       Synced                           prometheusrule.monitoring.coreos.com/boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00                              ConfigMap               boxever-public-service  filebeat-boxever-eks-multi-armed-bandit              Synced                           configmap/filebeat-boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00                             ServiceAccount           boxever-public-service  boxever-eks-multi-armed-bandit                       Synced                           serviceaccount/boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00  rbac.authorization.k8s.io        Role               boxever-public-service  boxever-eks-multi-armed-bandit                       Synced                           role.rbac.authorization.k8s.io/boxever-eks-multi-armed-bandit reconciled. role.rbac.authorization.k8s.io/boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00                                Service               boxever-public-service  boxever-eks-multi-armed-bandit                       Synced  Healthy                  service/boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00  monitoring.coreos.com      ServiceMonitor           boxever-public-service  boxever-eks-multi-armed-bandit                       Synced                           servicemonitor.monitoring.coreos.com/boxever-eks-multi-armed-bandit unchanged
2022-05-27T11:54:16+01:00  kubernetes-client.io       ExternalSecret           boxever-public-service  boxever-eks-multi-armed-bandit-ansible-secrets       Synced  Healthy                  externalsecret.kubernetes-client.io/boxever-eks-multi-armed-bandit-ansible-secrets unchanged
2022-05-27T11:54:16+01:00  policy                     PodDisruptionBudget      boxever-public-service  boxever-eks-multi-armed-bandit-pdb                   Synced                           poddisruptionbudget.policy/boxever-eks-multi-armed-bandit-pdb unchanged
<hangs here>

[ashutosh16]

@crenshaw-dev . Can you assign the issue to me, wants to give it a try

[lauravuo]

Any progress with this?

It seems that server side is not reporting authentication errors to client side: https://github.com/argoproj/argo-cd/blob/master/server/application/application.go#L1220:L1221

Once the token has grown old, sending an error to the stream would allow client to fail the call and user would be able to retry with a refreshed token.