Web Shell: response.WriteHeader on hijacked connection

[cite]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

When trying to open a WebShell for Pods running on the same Kubernetes cluster that ArgoCD is running on, the shell never opens, instead we get a blank window with a blinking cursor. At the same time, the following messages are logged:

time="2023-02-06T09:47:35Z" level=info msg="terminal session starting" appNamespace=argocd application=ahs-uploader container=ahs-uploader-gunicorn namespace=default podName=ahs-uploader-deployment-65db469959-z77v7 project=it-infrastructure userName=sfr@example.com
2023/02/06 09:47:35 http: response.WriteHeader on hijacked connection from github.com/argoproj/argo-cd/v2/server/application.(*terminalHandler).ServeHTTP (terminal.go:277)
2023/02/06 09:47:35 http: response.Write on hijacked connection from fmt.Fprintln (print.go:265)

For other clusters that ArgoCD has access too, this is working as intended (the log also doesn't say anything about hijacking there).

Version

argocd: v2.5.9+unknown
  BuildDate: 2023-01-30T14:20:38Z
  GitCommit: 
  GitTreeState: 
  GitTag: 2.5.9
  GoVersion: go1.19.5
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.5.9+e5f1194
  BuildDate: 2023-01-27T23:27:14Z
  GitCommit: e5f1194a6de78cc1124179a4c9bb1ae3484fb77d
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.3+g835b733
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

[rgdev]

I have the exact same issue on 2.6.0, also running on the same cluster as the pods i'm trying to open a shell on. Same messages about hijacked connections.

argocd: v2.6.0+acc554f.dirty
  BuildDate: 2023-02-07T01:29:54Z
  GitCommit: acc554f3d99010e0353b498a595844b30090556f
  GitTreeState: dirty
  GoVersion: go1.19.5
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.6.0+acc554f
  BuildDate: 2023-02-06T21:17:26Z
  GitCommit: acc554f3d99010e0353b498a595844b30090556f
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64

[rgdev]

After some testing it seems to occur only on alpine-based images. I tried setting exec.shells to something like ash,sh,bash to make sure ash was first but no dice.

[michizubi-SRF]

We experience the same issue running version 2.5.10.

[cite]

The issue is still present in 2.6.2.

[zoltan-szabo]

This feature would be something very nice to have, so I'm waiting to see it alive :-)

[raunoodem]

Having the same issue when upgrading from version 2.4.13 to 2.5.15. It was working fine with 2.4.13

[raunoodem]

Having the same issue when upgrading from version 2.4.13 to 2.5.15. It was working fine with 2.4.13

And also not working with version 2.6.6

[evenh]

Encountering the same running v2.6.7+5bcd846. I suspect this is due to the ClusterRole argocd-server is missing the create permission on pod/exec. See also these docs.

I'm applying this patch locally:

[
  {
    "op": "add",
    "path": "/rules/-",
    "value": {
      "apiGroups": [
        ""
      ],
      "resources": [
        "pods/exec"
      ],
      "verbs": [
        "create"
      ]
    }
  }
]

(using https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml with Kustomize)

[lhaussknecht]

Thanks @evenh! That was the missing piece.

[michizubi-SRF]

We did configure the permissions as mentioned in the docs but the issue remains.

[zoltan-szabo]

Same here, doesn't work. v2.6.7

[johnkk84]

I have same issue but my installation has delivered argocd through OpenShift Operator.

In logs I have:

2023/07/02 11:07:31 http: response.WriteHeader on hijacked connection from github.com/argoproj/argo-cd/v2/server/application.(*terminalHandler).ServeHTTP (terminal.go:245)
2023/07/02 11:07:31 http: response.Write on hijacked connection from fmt.Fprintln (print.go:285)

[Whisper40]

Encountering the same running v2.6.7+5bcd846. I suspect this is due to the ClusterRole argocd-server is missing the create permission on pod/exec. See also these docs.

I'm applying this patch locally:

[
  {
    "op": "add",
    "path": "/rules/-",
    "value": {
      "apiGroups": [
        ""
      ],
      "resources": [
        "pods/exec"
      ],
      "verbs": [
        "create"
      ]
    }
  }
]

(using https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml with Kustomize)

It seems not in our case, we already have this permission :

- apiGroups:
  - ""
  resources:
  - pods/exec
  verbs:
  - create