Getting 502 via Nginx Ingress despite reaching container

[greffgreff]

Environment

• ejabberd docker image: ejabberd/ecs:latest
• version: 23.04

Configuration

ejabberd.yml

```yaml hosts: - localhost loglevel: 4 log_rotate_size: 10485760 log_rotate_count: 1 certfiles: - /home/ejabberd/conf/server.pem ca_file: "/home/ejabberd/conf/cacert.pem" ## When using let's encrypt to generate certificates ##certfiles: ## - /etc/letsencrypt/live/localhost/fullchain.pem ## - /etc/letsencrypt/live/localhost/privkey.pem ## ##ca_file: "/etc/letsencrypt/live/localhost/fullchain.pem" default_db: sql sql_type: pgsql sql_server: sql_port: 5432 sql_database: sql_username: sql_password: auth_method: [sql] new_sql_schema: true listen: - port: 5222 ip: "::" module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s starttls_required: true - port: 5269 ip: "::" module: ejabberd_s2s_in max_stanza_size: 524288 - port: 5443 ip: "::" module: ejabberd_http tls: true request_handlers: "/admin": ejabberd_web_admin "/api": mod_http_api "/bosh": mod_bosh "/captcha": ejabberd_captcha "/upload": mod_http_upload "/ws": ejabberd_http_ws "/oauth": ejabberd_oauth - port: 5280 ip: "::" module: ejabberd_http request_handlers: "/admin": ejabberd_web_admin - port: 1883 ip: "::" module: mod_mqtt backlog: 1000 s2s_use_starttls: optional acl: local: user_regexp: "" loopback: ip: - 127.0.0.0/8 - ::1/128 - ::FFFF:127.0.0.1/128 admin: user: - "admin@localhost" xmp_service: ip: - ::FFFF:172.17.0.0/8 # not working access_rules: local: allow: local c2s: deny: blocked allow: all announce: allow: admin configure: allow: admin muc_create: allow: local pubsub_createnode: allow: local trusted_network: allow: loopback api_permissions: "console commands": from: - ejabberd_ctl who: all what: "*" "admin access": who: all # access: # allow: # acl: loopback # acl: admin # acl: xmp_service what: - "*" - "!stop" - "!start" "public commands": who: ip: 127.0.0.1/8 what: - status - connected_users_number shaper: normal: 1000 fast: 50000 shaper_rules: max_user_sessions: 10 max_user_offline_messages: 5000: admin 5000: all c2s_shaper: none: admin normal: all s2s_shaper: fast max_fsm_queue: 10000 acme: contact: "mailto:example-admin@example.com" ca_url: "https://acme-staging-v02.api.letsencrypt.org/directory" modules: mod_adhoc: {} mod_admin_extra: {} mod_announce: access: announce mod_avatar: {} mod_blocking: {} mod_bosh: {} mod_caps: {} mod_carboncopy: {} mod_client_state: {} mod_configure: {} mod_disco: {} mod_fail2ban: {} mod_http_api: {} mod_http_upload: put_url: https://@HOST@:5443/upload mod_last: {} mod_mam: db_type: sql assume_mam_usage: true default: always mod_mqtt: {} mod_muc: default_room_options: persistent: true public: false members_only: true mam: true allow_subscription: true db_type: sql access: - allow access_admin: - allow: admin access_create: muc_create access_persistent: muc_create access_mam: - allow mod_muc_admin: {} mod_offline: access_max_user_messages: max_user_offline_messages mod_ping: {} mod_privacy: {} mod_private: {} mod_proxy65: access: local max_connections: 5 mod_pubsub: access_createnode: pubsub_createnode plugins: - flat - pep force_node_config: storage:bookmarks: access_model: whitelist mod_push: {} mod_push_keepalive: {} mod_register: ip_access: trusted_network mod_roster: versioning: true mod_sip: {} mod_s2s_dialback: {} mod_shared_roster: {} mod_stream_mgmt: resend_on_timeout: if_offline mod_vcard: {} mod_vcard_xupdate: {} mod_version: show_os: false ```

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: chat-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    # nginx.org/websocket-services: ejabberd-service
    nginx.ingress.kubernetes.io/proxy-protocol: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/server-snippets: |
      location / {
        proxy_set_header Upgrade $http_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Connection "upgrade";
        proxy_cache_bypass $http_upgrade;
      }
spec:
  rules:
    - host:<domain>.net
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ejabberd-service
                port:
                  number: 5443

Ejabberd service

apiVersion: v1
kind: Service
metadata:
  name: ejabberd-service
spec:
  selector:
    app: ejabberd-app
  ports:
    - port: 5222
      targetPort: 5222
      name: ejabberd-5222
    - port: 5443
      targetPort: 5443
      name: ejabberd-5443
    - port: 5280
      targetPort: 5280
      name: ejabberd-5280

Ejabberd pod logs:

2023-05-25 11:42:49.176612+00:00 [info] (<0.886.0>) Accepted connection [::ffff:10.244.18.60]:34232 -> [::ffff:10.244.26.4]:5443

Ingress logs:

<ip> - - [25/May/2023:11:42:47 +0000] "GET /ws HTTP/1.1" 502 150 "-" "-" 258 0.007 [default-ejabberd-service-5443] [] <cluster ip>:5443, <cluster ip>:5443,<cluster ip>:5443 0, 0, 0 0.004, 0.001, 0.002 502, 502, 502 e6c80bc50e70a848a9e133efc9016997
2023/05/25 11:42:49 [error] 4952#4952: *316267 upstream prematurely closed connection while reading response header from upstream, client: <client ip>, server: <domain>.net, request: "GET /ws HTTP/1.1", upstream: "http://<cluster ip>:5443/ws", host: "<domain>.net"

Bug description

I have a kubernetes cluster deployed on an AKS instance. I am not able to connect completely to the /ws socket of port 5443 via an nginx ingress; while I am seeing that attempts to connect to /ws reaches the pod as per the ingress and ejabberd logs, I am still getting a 502 response back.

I have provided the configurations for the ingress with a sole endpoint to /ws along side the ejabberd service and the ejabberd configuration I am using and can provide more information as needed.

I was unsure wether to ask this question here or on the Kuberentes github.

[prefiks]

I see some mentions about proxy_protocol, if this triggers sending proxy_protocol frame at start, you also need to make ejabberd listener know about it by adding use_proxy_protocol: true in listener.

Also i see you are using 5443 port in ejabberd, and that is configured to have tls conneciton (aka it expect encrypted data, are your proxy sending tls or plain text?

[greffgreff]

I got tunnel visioned in the troubleshooting process and forgot expose port 5443 in the deployment file... Thanks anyways!