search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
16.82k stars 5.09k forks source link

Web Terminal: WebSocket connection to 'wss://example.com/sub/argo-cd/terminal' failed: #11954

Open danielsatanik opened 1 year ago

danielsatanik commented 1 year ago

I recently upgraded and now the Web Terminal is available, but I cannot get it to work with our nginx ingress. My original configuration looked as follows:

// argo-server.yaml
command:
- argocd-server
- --insecure
- --basehref 
- /sub/argo-cd
// ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite "(?i)/argo-cd(/|$)(.*)$" /$2 break;
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
  name: argocd-server-http-ingress
  namespace: argocd
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: argocd-server
            port:
              number: 80
        path: /argo-cd
        pathType: ImplementationSpecific

Important to know is everything works fine and the kubernetes cluster sits behind another external facing proxy that forwards all traffic on /sub to the cluster.

After granting exec rights, enabling it in the argo-cm.yaml and configuring the role I could access the UI elements for the webterminal but as soon as I click on the tab in the details section, the following is shown:

Browser Error:
WebSocket connection to 'wss://example.com/sub/argo-cd/terminal' failed:

Nginx Access Logs:
[...] "GET /argo-cd/terminal?[...] HTTP/1.0" 400 45 [...] [argocd-argocd-server-80] [...]

ArgoCD Server Logs:
[...] level=info msg="terminal session starting" [...]
[...] http: superfluous response.WriteHeader call from github.com/argoproj/argo-cd/v2/server/application.(*terminalHandler).ServeHTTP (terminal.go:237)

I have then added a server-snippet in the nginx ingress annotations according to some of the forums I have browsed to set the upgrade headers, that looked like this:

nginx.ingress.kubernetes.io/server-snippet: |
      location ~* "^/argo-cd/terminal" {
        proxy_set_header Upgrade $http_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Connection "upgrade";
        proxy_cache_bypass $http_upgrade;
      }

That led to:

Browser Error:
WebSocket connection to 'wss://example.com/sub/argo-cd/terminal' failed:

Nginx Access Logs:
[...] [error] 11189#11189: *60233189 open() "/usr/local/nginx/html/argo-cd/terminal" failed (2: No such file or directory) [...]
[...] "GET /argo-cd/terminal?[...] HTTP/1.0" 404 548 [...]

ArgoCD Server Logs:
empty

Which makes a lot of sense because now it won't be forwarded to the argo-server anymore, so I extended it by:

nginx.ingress.kubernetes.io/server-snippet: |
      location ~* "^/argo-cd/terminal" {
        # this line
        proxy_pass http://upstream_balancer;

        proxy_set_header Upgrade $http_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Connection "upgrade";
        proxy_cache_bypass $http_upgrade;
      }

Which led to:

Browser Error:
WebSocket connection to 'wss://example.com/sub/argo-cd/terminal' failed:

Nginx Access Logs:
[...] [crit] 11327#11327: *60238326 connect() to 0.0.0.1:80 failed (22: Invalid argument) while connecting to upstream [...] request: "GET /argo-cd/terminal?[...] HTTP/1.0", upstream: "http://0.0.0.1:80/argo-cd/terminal?[...]" [...]
[...] [warn] 11327#11327: *60238326 upstream server temporarily disabled while connecting to upstream [...] request: "GET /argo-cd/terminal?[...] HTTP/1.0", upstream: "http://0.0.0.1:80/argo-cd/terminal?[...]" [...]
[...] "GET /argo-cd/terminal?[...] HTTP/1.0" 502 552 [...]

ArgoCD Server Logs:
empty

Which still makes sense because I had my rewrite in the ingress which I have to add as well, which I did:

nginx.ingress.kubernetes.io/server-snippet: |
      location ~* "^/argo-cd/terminal" {
        proxy_pass http://upstream_balancer;
        # this line
        rewrite "(?i)/argo-cd(/|$)(.*)$" /$2 break;

        proxy_set_header Upgrade $http_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Connection "upgrade";
        proxy_cache_bypass $http_upgrade;
      }

Which returned the same error. So I looked into the nginx.conf of the ingress and added a bunch of other stuff:

nginx.ingress.kubernetes.io/server-snippet: |
      location ~* "^/argo-cd/terminal" {
        # this block
        set $namespace      "argocd";
        set $service_name   "argocd-server";
        set $service_port   "80";
        set $location_path  "/argo-cd/terminal";
        set $proxy_upstream_name "argocd-argocd-server-80";
        set $proxy_host          $proxy_upstream_name;
        set $pass_access_scheme  $scheme;
        set $pass_server_port    $server_port;
        set $best_http_host      $http_host;
        set $pass_port           $pass_server_port;
        set $proxy_alternative_upstream_name "";

        proxy_pass http://upstream_balancer;
        rewrite "(?i)/argo-cd(/|$)(.*)$" /$2 break;

        proxy_set_header Upgrade $http_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Connection "upgrade";
        proxy_cache_bypass $http_upgrade;
      }

Now it goes through to the argo-server but still throws a 400.

Browser Error:
WebSocket connection to 'wss://example.com/sub/argo-cd/terminal' failed:

Nginx Access Logs:
[...] "GET /argo-cd/terminal?[...] HTTP/1.0" 400 45 [...] [argocd-argocd-server-80] [...]

ArgoCD Server Logs:
[...] level=info msg="terminal session starting" [...]
[...] http: superfluous response.WriteHeader call from github.com/argoproj/argo-cd/v2/server/application.(*terminalHandler).ServeHTTP (terminal.go:237)

And I'm back at the beginning. The error message is exactly the same. I looked closer into the auto-generated part of the nginx.conf and it even allows websockets so the problem must be somewhere else. Here is a small excerpt from the location block ingress generates:

location ~* "^/argo-cd/terminal" {
  set $namespace      "argocd";
  set $ingress_name   "argocd-server-http-ingress";
  set $service_name   "argocd-server";
  set $service_port   "80";
  set $location_path  "/argo-cd";
  rewrite_by_lua_block {
    lua_ingress.rewrite({
      force_ssl_redirect = false,
      ssl_redirect = false,
      force_no_ssl_redirect = false,
      use_port_in_redirects = false,
    })
    balancer.rewrite()
    plugins.run()
  }

  # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
  # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
  # other authentication method such as basic auth or external auth useless - all requests will be allowed.
  access_by_lua_block {
  }

  header_filter_by_lua_block {
    lua_ingress.header()
    plugins.run()
  }

  body_filter_by_lua_block {
  }

  log_by_lua_block {
    balancer.log()
    monitor.call()
    plugins.run()
  }
  port_in_redirect off;
  set $balancer_ewma_score -1;
  set $proxy_upstream_name "argocd-argocd-server-80";
  set $proxy_host          $proxy_upstream_name;
  set $pass_access_scheme  $scheme;
  set $pass_server_port    $server_port;
  set $best_http_host      $http_host;
  set $pass_port           $pass_server_port;
  set $proxy_alternative_upstream_name "";
  client_max_body_size                    1m;
  proxy_set_header Host                   $best_http_host;
  # Pass the extracted client certificate to the backend
  # Allow websocket connections
  proxy_set_header                        Upgrade           $http_upgrade;
  proxy_set_header                        Connection        $connection_upgrade;

  proxy_set_header X-Request-ID           $req_id;
  proxy_set_header X-Real-IP              $remote_addr;
  proxy_set_header X-Forwarded-For        $remote_addr;
  proxy_set_header X-Forwarded-Host       $best_http_host;
  proxy_set_header X-Forwarded-Port       $pass_port;
  proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
  proxy_set_header X-Scheme               $pass_access_scheme;    # Pass the original X-Forwarded-For
  proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
  # mitigate HTTPoxy Vulnerability
  # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

  proxy_set_header Proxy                  "";
  # Custom headers to proxied server
  proxy_connect_timeout                   5s;
  proxy_send_timeout                      60s;
  proxy_read_timeout                      60s;
  proxy_buffering                         off;
  proxy_buffer_size                       4k;
  proxy_buffers                           4 4k;
  proxy_max_temp_file_size                1024m;
  proxy_request_buffering                 on;
  proxy_http_version                      1.1;
  proxy_cookie_domain                     off;
  proxy_cookie_path                       off;
  # In case of errors try the next upstream server before returning an error

  proxy_next_upstream                     error timeout;
  proxy_next_upstream_timeout             0;
  proxy_next_upstream_tries               3;
  rewrite "(?i)/argo-cd(/|$)(.*)$" /$2 break;
  proxy_pass http://upstream_balancer;
  proxy_redirect                          off;
}

So where exactly can I make changes to make the setting work?

Issues that haven't helped:

gururajumoe commented 1 year ago

+1

inghelsyorick commented 8 months ago

+1 Put exec.enabled: true, used admin account and also tried SSO with the correct RBAC rights. Argo is behind GCP ingress, and we also use Istio service mesh. Running insecure flag for server.

WebSocket connection to 'wss://URL/terminal?pod=test-64469d7d7f-k4kv5&container=app&appName=client&appNamespace=argocd&projectName=dev&namespace=client' failed:

Running Argo v2.9.0

Any updates for this issue?

stevenlafl commented 5 months ago

+1 this problem occurs with Traefik periodically, other times there is an 101 Upgrade response followed by an outgoing message {"operation":"resize","cols":95,"rows":47}, and ultimately resulting in Connection Closed: 1006

Senderman commented 4 months ago

+1 for {"operation":"resize","cols":121,"rows":43} when opening terminal here's my nginx config:

location / {
        proxy_pass http://10.108.252.116:80/; <--- ClusterIP of ArgoCD Service
    proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }