Websocket error

[dexma-dev]

Hello everybody,

i am trying to install the mattermost team edition chart on my cluster but I am running into some websocket issues. I am experiencing the same issues as described here.

Is this an issue of the helm chart, the docker container or the nginx ingress?

{"timestamp":"2022-03-31 21:10:41.886 Z","level":"error","msg":"Failed to upgrade websocket connection.","caller":"web/context.go:105","path":"/api/v4/websocket","request_id":"a1us537gp3b7xxyoawiuksyrmr","ip_addr":"xxxxx","user_id":"xxxxxx","method":"GET","err_where":"connect","http_code":500,"err_details":"websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header"}

UPDATE: Please keep in mind that I am on kubernetes version 1.23.1 and i am therefore using the networking.k8s.io/v1 ingress api. I am not sure this issue is related to that. I made the same changes to the ingress as described in #250. Everything else is working as expected.

---

Kubernetes version: v1.23.1 Ingress: Nginx Ingress apiVersion: networking.k8s.io/v1

[dexma-dev]

Is this related to #280?

[agnivade]

@gabrieljackson / @Szymongib - Any thoughts on this? It looks like the nginx config isn't upgrading the websocket connection.

[Szymongib]

and i am therefore using the networking.k8s.io/v1 ingress api

Do you create the ingress manually then? Could you share it here so we can investigate?

[dexma-dev]

Thank you for your time and help! I cloned the repo and changed the ingress definition inside the helm chart (team-edition). Here is the resulting ingress definition:

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: mattermost-mattermost-team-edition
  namespace: gitlab
  labels:
    app.kubernetes.io/instance: mattermost
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mattermost-team-edition
    helm.sh/chart: mattermost-team-edition-6.4.1
  annotations:
    acme.cert-manager.io/http01-edit-in-place: 'true'
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
    meta.helm.sh/release-name: mattermost
    meta.helm.sh/release-namespace: gitlab
    nginx.ingress.kubernetes.io/client-max-body-size: 1G
    nginx.ingress.kubernetes.io/proxy-body-size: 1G 
spec:
  tls:
    - hosts:
        - mattermost.domain.com
      secretName: secret-name
  rules:
    - host: mattermost.domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: mattermost-team-edition
                port:
                  number: 8065
status:
  loadBalancer: {}

After several failed attempts i also tried to add a second ingress especially for the upgrade of the connection:

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: mattermost-mattermost-team-edition-header
  namespace: gitlab
  labels:
    app.kubernetes.io/instance: mattermost
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mattermost-team-edition
    helm.sh/chart: mattermost-team-edition-6.4.1
  annotations:
    acme.cert-manager.io/http01-edit-in-place: 'true'
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
    meta.helm.sh/release-name: mattermost
    meta.helm.sh/release-namespace: gitlab
    nginx.ingress.kubernetes.io/client-max-body-size: 1G
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Upgrade: $http_upgrade";
      more_set_headers "Connection: 'upgrade'";
      proxy_set_header "Upgrade: $http_upgrade";
      proxy_set_header "Connection: 'upgrade'";
    nginx.ingress.kubernetes.io/proxy-body-size: 1G
spec:
  tls:
    - hosts:
        - mattermost.domain.com
      secretName: secret-name
  rules:
    - host: mattermost.domain.com
      http:
        paths:
          - path: /api/v[0-9]+/(users/)?websocket$
            pathType: Prefix
            backend:
              service:
                name: mattermost-team-edition
                port:
                  number: 8065
status:
  loadBalancer: {}

[Szymongib]

Thanks! Could you also share some details about your setup? Are you running on some Cloud? Perhaps the Nginx config/chart values could be helpful too.

[dexma-dev]

I am running Bare Metal with the following configurations:

Kubernetes Version: v1.23.1

---

NGINX Ingress: v2.1.1 Installation Type: Manifests

No special configuration. I followed the installation guide which can be found here.

---

MetalLB LoadBalancer: v0.12.1 Installation Type: Manifests

No special configuration here either. Followed the installation guide which can be found here.

---

All of this was running perfectly on kubernetes version 1.19 a couple of weeks ago. I upgraded to the newest version, therefore i am pretty sure it has something to do with the new ingress api or the newer version of the ingress controller.

@Szymongib any ideas where the problem is coming from?

[Szymongib]

Thanks for the details @cneumaier. So is the issue only with websockets? Can you access Mattermost at all?

[dexma-dev]

@Szymongib Everything else is working as expected. I am having only issues with the websockets

[Szymongib]

That is just a blind guess but I think the Ingress controller you are using is a different thing although based on Nginx too.

I think nginx.ingress.kubernetes.io annotation is for https://github.com/kubernetes/ingress-nginx and the link you sent is https://github.com/nginxinc/kubernetes-ingress/ which according to docs uses nginx.org annotation. I have no experience with the latter so it is hard for me to say if it is configured properly.

[dexma-dev]

You could be onto something there. Let me investigate further and maybe use the „normal“ nginx ingress. I will report back

[dexma-dev]

@Szymongib your guess was on point!

I removed my ingress controller and I installed the kubernetes/ingress-nginx version, which resolved the websocket error. I did not investigate any further why the other ingress controller caused the issue.

Thank you very much for your help!