k8s-dashboard exec is giving black screen intermittently & sometime with error - Sending Error: Error: http status 404

[shnigam2]

Issue - When trying exec into pod through kubernetes dashboard , intermittently giving black screen with error Sending Error: Error: http status 404

EKS version - 1.26.7 k8s-dashboard version - 6.0.8 Openunison version -

chart: orchestra-login-portal-argocd
        targetRevision: 2.3.50

Values for dashboard

            dashboard:
              namespace: "kubernetes-dashboard"
              cert_name: "kubernetes-dashboard-certs"
              label: "k8s-app=kubernetes-dashboard"
              service_name: kubernetes-dashboard
              require_session: true

[shnigam2]

@mlbiam Request to please check

[mlbiam]

what versions of the containers are you running?

[shnigam2]

@mlbiam please find log from openunison pod for version

k logs openunison-openunison-d4696498b-4cpns -n openunison |head
/usr/local/openunison/work/webapp/WEB-INF/lib/*:/usr/local/openunison/work/webapp/WEB-INF/classes:/tmp/quartz
[2024-03-16 00:05:25,763][main] INFO  OpenUnisonOnUndertow - Starting OpenUnison on Undertow 1.0.37-2023072501

[shnigam2]

@mlbiam Could you please take a look, we are using below version and corresponding images. We are facing issues with dashboard exec intermittently with 2 replicas of dashboard, But we have tried keeping 1 replica of dashboard it worked fine then. How could we fix dashboard exec issue by keeping 2 replicas of dashboard.

chart: orchestra-login-portal-argocd
        targetRevision: 2.3.50

            impersonation:
              use_jetstack: true
              explicit_certificate_trust: true
              ca_secret_name: secret-ca

Dashboard ingress values:-

              openunison-dashboard:
                annotations:
                  kubernetes.io/ingress.class: nginx
                  certmanager.k8s.io/cluster-issuer: letsencrypt
                  nginx.ingress.kubernetes.io/server-alias: {{ metadata.annotations.environment }}-dashboard-{{ metadata.annotations.name }}.company.com
                  nginx.ingress.kubernetes.io/upstream-vhost: {{ metadata.annotations.environment }}-dashboard-{{ metadata.annotations.name }}.company.com
                  nginx.ingress.kubernetes.io/affinity: cookie
                  nginx.ingress.kubernetes.io/backend-protocol: https
                  nginx.ingress.kubernetes.io/secure-backends: 'true'
                  nginx.ingress.kubernetes.io/session-cookie-hash: sha1
                  nginx.ingress.kubernetes.io/session-cookie-name: openunison-orchestra
                  nginx.org/ssl-services: openunison-orchestra
                hosts:
                - name: dashboard.{{ metadata.annotations.fqdn }}
                  service:
                  - externalPort: 443
                    servicename: openunison-orchestra
                    path: "/"
                    pathType: ImplementationSpecific
                  tls: true
                namespace: openunison

[shnigam2]

By inspect the response we are getting below error :-

failed: WebSocket is closed before the connection is established.

[shnigam2]

@mlbiam Please suggest on this

[mlbiam]

I've been unable to reproduce this issue with the latest versions on EKS. Please try upgrading to openunison 1.0.40 (the latest) and see if the issue persists.

[shnigam2]

@mlbiam Still facing this issue with 1.0.40 image as well

[shnigam2]

@mlbiam Issue if you try multiple times then may be you can able to see this issue intermittently. Please try with 2 replica of dashboard, we are facing this issue with 2 replicas of dashboard only.

Tested on EKS version - v1.28.5-eks-5e0fdde

[mlbiam]

we are facing this issue with 2 replicas of dashboard only.

I see. The issue is likely a problem with session stickyness. OpenUnison doesn't do any load balancing. It's entirely up to external services (such as a virtual ip or some other load balancer). In this instance, OpenUnison is communicating to a Kubernetes Service which doesn't support any kind of session stickyness. If you want to support HA apps inside of the cluster, you'll need something that will do layer 7 load balancing. The easiest way I can think to do this is to deploy ingress nginx with a ClusterIP and a unique IngressClass. Then you can define an Ingress object for your internal IngressClass, and a Service that targets the internal ingress nginx pods instead of the dashboard pods.

honestly, I'm not sure it's worth the effort. The dashboard is stateless generally (outside of this one use case). Running more then one replica isn't buying you much.

Since this is outside of OpenUnison, closing.