Critical Mosquitto Bug: mixing up payloads from different topics

[canique]

If a message has expired, it can happen that Mosquitto writes a wrong payload to some random topic when restarting.

The messages are published with MQTTv5. Mosquitto version: 2.0.12 Retain flags are set for all messages. The message expiry interval is set for only 1 message.

The result is that you e.g. end up with a json payload in a topic that you have only published plaintext messages to.

I would consider this bug as extremely severe. A broker that mixes up messages ain't no broker.

[Daedaluz]

Have this occurred multiple times? Can you replicate the issue consistently?

I tried this by

• Setup a clean broker with storage configured.
• publish a few retained messages for i in $(seq 0 1 10); do mosquitto_pub -V mqttv5 -t "topic/$i" -m "$i" -r ; done
• Have one mosquitto_sub -V mqttv5 -v -t "#" -q 2 -i subscriber -c
• publish one message with expiry interval set to 60 mosquitto_pub -V mqttv5 -t 'topic/expiry' -m 'hello_world' -r -D PUBLISH message-expiry-interval 60
• repeatedly restart broker and subscriber until message is expired and gone.

I also tried

• publish the expiry message
• stop broker
• wait till after message should be expired
• start broker

but i couldn't see any messages with mixed up payloads

[canique]

It has happened multiple times. I cannot replicate it anymore right now.

First I was publishing on a remote bridge. As I had specified a max lifetime for messages, the bridge automatically added a message expiry interval to everything I was publishing too. These messages would then be forwared to Mosquitto. Then after a while json messages would show up in topics on my local mosquitto that were supposed to have plaintext data. I could reproduce it back then by using https://github.com/mqttjs/MQTT.js/#publish to publish data with a short expiry interval and then restarting mosquitto.

[ralight]

@canique Are you able to share any more specific details that might help? For example, your broker configuration, which topics you are publishing to - anything at that you might have done that we don't realise would be helpful.

[canique]

The bug happened again today, but I cannot tell exactly what triggered the bug.

I've shut down the WiFi of the Mosquitto node once or twice, and I've shutdown the remote HiveMQ broker twice. One of these events must have caused mixing up topics again.

I've considered the possibility that the bug is on the HiveMQ side, and not on the mosquitto side. But it is very improbable. Reason: In my test configuration Mosquitto is set up with ACLs that disallow the remote broker to write on certain topics. Still I've got wrong data on exactly those topics. So it must have been Mosquitto itself that wrote the data on these topics.

For now I can only say that it has something to do with reconnection...

[canique]

Analysis:

When the bug first occurred, it happened on topics that were IN/OUT topics, that is: on topics that were synchronized in both directions between Mosquitto and HiveMQ.

The last time the bug occurred, it happend on a topic that is only written to by Mosquitto (an OUT topic) and then transferred to the remote HiveMQ broker. The interesting part is: The wrong data landed on the remote broker, but not on the Mosquitto broker. So when I connect to the remote broker, I can see the wrong retained message. But when I connect to Mosquitto, I don't see it.

Which -as of now- leads me to the conclusion, that: 1) on the first occurence of this bug, HiveMQ might have written the wrong data and then synchronized it to Mosquitto 2) on the last occurence of this bug, HiveMQ again wrote the wrong data but this time on a topic that is not synchronized.

I'm still not sure, but this is the current state.

[canique]

regarding reproduceability:

I could reproduce the bug multiple times now by restarting HiveMQ. But a subcondition must be met, it seems. A client must be connected to HiveMQ while restarting. It seems only then will the bug occur.

1) MQTT client connects to HiveMQ 2) Restart HiveMQ 3) MQTT client tries to reconnect in 2s intervals, eventually reconnects 4) Mosquitto <-> HiveMQ bridge is re-established 5) topics get overwritten with data from other topics on the HiveMQ side. It's unclear whether this data comes from Mosquitto or from some HiveMQ bug

[canique]

Good news:

After some more testing, I came to the conclusion that the bug must be on the HiveMQ side. And indeed, that's where the bug was: https://github.com/hivemq/hivemq-community-edition/issues/306

The issue has been fixed in the nightly HiveMQ builds. It has nothing to do with Mosquitto. Sorry for blaming Mosquitto.

[ralight]

No problem, I've glad you've got it sorted.