[Bug]: mq disconnected due to protocol error

[devudopw]

Contact Details

No response

What happened?

After created network, join some nodes, the mq started get some error. Here is the docker-compose

version: "3.4"

services:
  netmaker:
    container_name: netmaker
    image: gravitl/netmaker:v0.16.0
    cap_add:
    - NET_ADMIN
    - NET_RAW
    - SYS_MODULE
    sysctls:
    - net.ipv4.ip_forward=1
    - net.ipv4.conf.all.src_valid_mark=1
    - net.ipv6.conf.all.disable_ipv6=0
    - net.ipv6.conf.all.forwarding=1
    restart: always
    volumes:
   #- dnsconfig:/root/config/dnsconfig
    - sqldata:/root/data
    - shared_certs:/etc/netmaker
    environment:
      SERVER_NAME: "broker.nm.domain"
      SERVER_HOST: "188.188.188.188"
      SERVER_API_CONN_STRING: "api.nm.domain:443"
       #COREDNS_ADDR: "188.188.188.188"
      DNS_MODE: "off"
      SERVER_HTTP_HOST: "api.nm.domain"
      API_PORT: "8081"
      CLIENT_MODE: "on"
      MASTER_KEY: "6832d803b1ccdafb1b858471fca3312c"
      CORS_ALLOWED_ORIGIN: "*"
      DISPLAY_KEYS: "on"
      DATABASE: "rqlite"
      SQL_CONN: "http://rqlite.nm.domain:4001"
      NODE_ID: "netmaker-server-1"
      MQ_HOST: "mq"
      MQ_PORT: "443"
      MQ_SERVER_PORT: "1883"
      HOST_NETWORK: "off"
      VERBOSITY: "1"
      MANAGE_IPTABLES: "off"
     #PORT_FORWARD_SERVICES: "dns"
    ports:
    - "51821:51821/udp"
    expose:
    - "8081"
    labels:
    - traefik.enable=true
    - traefik.http.routers.netmaker-api.entrypoints=websecure
    - traefik.http.routers.netmaker-api.rule=Host(`api.nm.domain`)
    - traefik.http.routers.netmaker-api.service=netmaker-api
    - traefik.http.services.netmaker-api.loadbalancer.server.port=8081
  netmaker-ui:
    container_name: netmaker-ui
    image: gravitl/netmaker-ui:v0.16.0
    depends_on:
    - netmaker
    links:
    - "netmaker:api"
    restart: always
    environment:
    BACKEND_URL: "https://api.nm.domain"
    expose:
    - "80"
    labels:
    - traefik.enable=true
    - traefik.http.middlewares.nmui-security.headers.accessControlAllowOriginList=*.nm.domain
    - traefik.http.middlewares.nmui-security.headers.stsSeconds=31536000
    - traefik.http.middlewares.nmui-security.headers.browserXssFilter=true
    - traefik.http.middlewares.nmui-security.headers.customFrameOptionsValue=SAMEORIGIN
    - traefik.http.middlewares.nmui-security.headers.customResponseHeaders.X-Robots-Tag=none
    - traefik.http.middlewares.nmui-security.headers.customResponseHeaders.Server= # Remove the server name
    - traefik.http.routers.netmaker-ui.entrypoints=websecure
    - traefik.http.routers.netmaker-ui.middlewares=nmui-security@docker
    - traefik.http.routers.netmaker-ui.rule=Host(`dashboard.nm.domain`)
    - traefik.http.routers.netmaker-ui.service=netmaker-ui
    - traefik.http.services.netmaker-ui.loadbalancer.server.port=80
  traefik:
    image: traefik:2.9
    container_name: traefik
    command:
    - "--certificatesresolvers.http.acme.email=abc.com"
    - "--certificatesresolvers.http.acme.storage=/letsencrypt/acme.json"
    - "--certificatesresolvers.http.acme.tlschallenge=true"
    - "--entrypoints.websecure.address=:443"
    - "--entrypoints.websecure.http.tls=true"
    - "--entrypoints.websecure.http.tls.certResolver=http"
    - "--log.level=INFO"
    - "--providers.docker=true"
    - "--providers.docker.exposedByDefault=false"
    - "--serverstransport.insecureskipverify=true"
    restart: always
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - traefik_certs:/letsencrypt
    ports:
    - "443:443"
  mq:
    container_name: mq
    image: eclipse-mosquitto:2-openssl
    depends_on:
      - netmaker
    restart: unless-stopped
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf
      - mosquitto_data:/mosquitto/data
      - mosquitto_logs:/mosquitto/log
      - shared_certs:/mosquitto/certs
    expose:
      - "8883"
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.mqtts.rule=HostSNI(`broker.nm.domain`)
      - traefik.tcp.routers.mqtts.tls.passthrough=true
      - traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883
      - traefik.tcp.routers.mqtts.service=mqtts-svc
      - traefik.tcp.routers.mqtts.entrypoints=websecure
volumes:
  traefik_certs: {}
  shared_certs: {}
  sqldata: {}
  dnsconfig: {}
  mosquitto_data: {}
  mosquitto_logs: {}

Version

v0.16.0

What OS are you using?

Linux

Relevant log output

mq             | 1665048154: New connection from 192.168.32.4:48064 on port 8883.
mq             | 1665048154: Client <unknown> disconnected due to protocol error.
mq             | 1665048156: New connection from 192.168.32.4:48070 on port 8883.
mq             | 1665048156: Client <unknown> disconnected due to protocol error.
mq             | 1665048158: New connection from 192.168.32.4:34960 on port 8883.
mq             | 1665048158: Client <unknown> disconnected due to protocol error.
mq             | 1665048160: New connection from 192.168.32.4:34976 on port 8883.
mq             | 1665048160: Client <unknown> disconnected due to protocol error.
mq             | 1665048162: New connection from 192.168.32.4:34988 on port 8883.
mq             | 1665048162: Client <unknown> disconnected due to protocol error.
mq             | 1665048164: New connection from 192.168.32.4:35002 on port 8883.
mq             | 1665048164: Client <unknown> disconnected due to protocol error.
mq             | 1665048166: New connection from 192.168.32.4:35014 on port 8883.
mq             | 1665048166: Client <unknown> disconnected due to protocol error.
mq             | 1665048168: New connection from 192.168.32.4:45368 on port 8883.
mq             | 1665048168: Client <unknown> disconnected due to protocol error.
mq             | 1665048170: New connection from 192.168.32.4:45374 on port 8883.
mq             | 1665048170: Client <unknown> disconnected due to protocol error.
mq             | 1665048172: New connection from 192.168.32.4:45376 on port 8883.

Contributing guidelines

• [X] Yes, I did.

[mattkasun]

https://gist.github.com/mattkasun/face2a7c1f32031a2126ff7243caad12

[devudopw]

https://gist.github.com/mattkasun/face2a7c1f32031a2126ff7243caad12

No help from the gist. Found the one mentioned by adding label to mq service, and I tried remove the label traefik.tcp.routers.mqtts.rule and port binding, the error has gone now.

mq:
    container_name: mq
    image: eclipse-mosquitto:2-openssl
    depends_on:
      - netmaker
    restart: unless-stopped
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf
      - mosquitto_data:/mosquitto/data
      - mosquitto_logs:/mosquitto/log
      - shared_certs:/mosquitto/certs
    #expose: # remove port binding
    #  - "8883"
    labels:
      - traefik.enable=true
      #- traefik.tcp.routers.mqtts.rule=HostSNI(`broker.nm.domain`) # remove this label
      - traefik.tcp.routers.mqtts.tls.passthrough=true
      - traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883
      - traefik.tcp.routers.mqtts.service=mqtts-svc
      - traefik.tcp.routers.mqtts.entrypoints=websecure