Docker Swarm socket error on broker

[zombielinux]

I have the latest docker image pulled from here: https://hub.docker.com/_/eclipse-mosquitto

I have a docker swarm set up, and the container defined with docker-compose (see attached)

version: '3.3'
services:
  mqtt:
    image: eclipse-mosquitto:latest
    ports:
     - 1883:1883
     - 9001:9001
    volumes:
     - mqtt-config:/mosquitto/config
     - mqtt-data:/mosquitto/data
     - mqtt-log:/mosquitto/log
    networks:
     - default
    logging:
      driver: json-file
    deploy:
      placement:
        constraints:
         - node.role==worker

networks:
  default:
    driver: overlay

volumes:
  mqtt-config:
    driver: local
    driver_opts:
      device: :/DockerDisks/mqtt/config
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs      
  mqtt-data:
    driver: local
    driver_opts:
      device: :/DockerDisks/mqtt/data
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs      
  mqtt-log:
    driver: local
    driver_opts:
      device: :/DockerDisks/mqtt/log
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs            

I have my swarm fronted by HAProxy to load balance between my ingress nodes.

I have many devices pointing to the HAProxy address (dockercluster.fqdn.org), and the data mostly seems to get there.

I get frequent disconnects claiming socket error. A snippit of my log is attached.

1603920309: New connection from 10.0.0.7 on port 1883.,
1603920323: Socket error on client mqttjs_3933e606, disconnecting.,
1603920301: New connection from 10.0.0.3 on port 1883.,
1603920294: Socket error on client <unknown>, disconnecting.,
1603920301: New client connected from 10.0.0.3 as mqttjs_bb254027 (p2, c1, k60, u'will').,
1603920302: Client mqttjs_bb254027 disconnected.,
1603920307: New connection from 10.0.0.3 on port 1883.,
1603920307: Socket error on client <unknown>, disconnecting.,
1603920307: Socket error on client DVES_018F73, disconnecting.,
1603920308: New connection from 10.0.0.3 on port 1883.,
1603920308: Socket error on client DVES_BF6989, disconnecting.,
1603920308: New client connected from 10.0.0.3 as DVES_018F73 (p2, c1, k30, u'will').,
1603920309: New client connected from 10.0.0.7 as DVES_BF6989 (p2, c1, k30, u'DVES_USER').,
1603920310: New connection from 10.0.0.3 on port 1883.,
1603920309: Socket error on client DVES_BEE459, disconnecting.,
1603920310: New client connected from 10.0.0.3 as DVES_BEE459 (p2, c1, k30, u'DVES_USER').,
1603920312: New connection from 10.0.0.232 on port 1883.,
1603920312: Socket error on client <unknown>, disconnecting.,
1603920314: New connection from 10.0.0.3 on port 1883.,
1603920314: Socket error on client <unknown>, disconnecting.,
1603920323: Socket error on client mqttjs_9455fae5, disconnecting.,
1603920323: Socket error on client mqttjs_1c6080e3, disconnecting.,
1603920324: New connection from 10.0.0.7 on port 1883.,
1603920293: New client connected from 10.0.0.7 as mqttjs_3933e606 (p2, c1, k60, u'will').,
1603920294: New connection from 10.0.0.7 on port 1883.

I do not have authentication turned on.

It appears multiple devices are trying to use the same IP address to connect from and its kicking a previous connection off (maybe?)

I'm trying to wrap my head around what could be going on here so I can get some stability in my devices.

mosquitto_sub -h dockercluster.fqdn.org -t '#' doesn't seem to show anything odd either.

[jpmens]

The kicking of connections will not likely be due to IP addresses, but to _clientID_s. Do you have clients connecting simultaneously with the same ID? That's a reason for Mosquitto to kick a client.

[zombielinux]

I don't think so.

Most of my devices are running the tasmota firmware (8.5.1 to be specific). I'm pretty sure they default to DVES_$macAddress and I haven't changed them.

I do have one bit of JS that just subscribes.

And there is homeassistant as well. Overall, not much on the broker at all.

[ralight]

You might get more information with debug logging, or if you're willing to try the develop branch then you get a wider variety of disconnection messages.

To do that:

git clone https://github.com/eclipse/mosquitto
cd mosquitto
git checkout develop
make localdocker

At the end of the process you will have an eclipse-mosquitto:local image.