Unstable connection to EMQX broker

[Audiobuzz]

What happened?

Recently moved over from using mosquitto to using a kubernetes hosted EMQX instance.

ZIgbee2mqtt seems unable to maintain a stable connection to the emqx broker whereas other mqtt clients are working fine with it.

z2m continues as if it is still connected (logs indicate it is still publishing) but the broker doesn't log receipt of anything and the other clients never receive the topic updates. The emqx has log entries for "congested connection" only for the z2m client. This error indicates the TCP buffer is full from the client not reading the socket.

Eventually the broker reaches the keepalive timeout and kicks the client off and closes the TCP connection. Z2m doesn't notice this immediately and eventually hits the TCP timeout and reestablishes the connection. Topic data flows for about 5-10 seconds before locking up again.

What did you expect to happen?

When moving to the new broker, I expected z2m to pub/sub correctly and continue to work the same as other clients have done.

How to reproduce it (minimal and precise)

start z2m and let it connect to an EMQX broker. Observe topic data initially and then stopping soon after. Eventually z2m ends up reconnecting and starting the cycle again.

Zigbee2MQTT version

1.35.3-dev commit: 7c97902

Adapter firmware version

20220219

Adapter

sonoff zigbee 3.0 usb dongle plus

Setup

Raspberry pi 3b with Ubuntu 20.04, z2m in docker-compose

Debug log

log.txt emqx@emqx-core-c8978cf8-0.emqx-headless.emqx-operator-system.svc.cluster.local-trace_zigbee2mqtt_2024-02-25.log

[mundschenk-at]

Did you adjust the TCP network settings for your EMQX nodes according to the documented examples?

[andersjohansson3]

I seem to have the same issue as @Audiobuzz. also recently moved both to EMQX and zigbee2mqtt. zigbee2mqtt keeps dropping the connection to EMQX broker, filling up the log with Zigbee2MQTT:error 2024-03-07 23:07:32: MQTT error: read ECONNRESET Zigbee2MQTT:info 2024-03-07 23:07:33: Connected to MQTT server Zigbee2MQTT:info 2024-03-07 23:07:33: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload '{"state":"online"}' Zigbee2MQTT:info 2024-03-07 23:07:35: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload '{"state":"online"}' Zigbee2MQTT:error 2024-03-07 23:07:35: MQTT error: write ECONNRESET

@mundschenk-at can you point to these documented examples?

[Audiobuzz]

Did you adjust the TCP network settings for your EMQX nodes according to the documented examples?

I would have thought that TCP settings would affect both EMQX and mosquitto but I'm happy to give them a shot if you can point me to the required settings?

[andersjohansson3]

I managed to get my zigbee2mqtt working again after a couple of re-installations, removing all config files and putting backup db and configuration back. What seemed to "fix" my issue for now was removing the locally stored icons from the configuration file. But I don't know how that could have been related with mqtt connection...

[convicte]

I've faced very similar issues over the last to days, but thus far was unable to resolve them: The coordinator being down for almost a day completely deteriorated my network in the process. I seem to have eliminated all confounders, becuase both ESPHome, NodeRed, Nuki locks, etc. logged into the new coordinator under the same credentials as the old one just fine. Z2m is constantly failing to connect to MQTT with an ECONNRESET error. I maintained network sessions and passwords across MQTT brokers and everything but Z2m transitions seamlessly.

I'll make an attempt to remove the local icons, though I struggle understanding how this would help as well.

I'd greatly appreciate assistance as repairing 60+ devices is no fun....

[blauhorn]

I am experiencing the same problem. z2m runs on a raspberry pi, the MQTT broker on another one. Restarting the z2m -pi brought improvement.

[convicte]

That is interesting to hear there is actually variability between deployments. For me EMQX is running inside HA as an addon and both are a part of HA VM in Unraid. I can't get Z2m to log in to EMQX MQTT at all - as seen above.

I wonder, how would one give visibility to this issue over 1200 others currently open, without annoying any of the devs?

[mpuff]

I moved also to emqx broker, now I get error at startup: zigbee2mqtt z2m: MQTT error: write EPIPE

I only created a mqtt user in emqx like I had it before in mosquitto. i will now change back to mosquitto and check if zigbee2mqtt is working again, good that Matt is running in docker in my setup

[convicte]

Same issue persists from the earlier reports. I transitioned like this for testing multiple times and got the same error every time.

Works in mosquito and fails to work in emqx.

[mpuff]

zigbee2mqtt is starting with mosquitto again

[bartlebi]

I can confirm that removing the icon lines from devices.yaml solves the problem.