search

peterhinch / micropython-mqtt

A 'resilient' asynchronous MQTT driver. Recovers from WiFi and broker outages.
MIT License
550 stars 117 forks source link

why republish with new PID? #39

Closed tve closed 2 years ago

tve commented 4 years ago

In the mqtt_as readme you write:

The publication is then attempted again as a new message with a different PID. (The new PID proved necessary for Mosquitto to recognise the message).

Can you describe why a new PID was necessary? If the client reconnects it would be best not to use a new PID to avoid unnecessary duplication of the message if the broker had actually received it. Is there a bug in mosquitto or what did you observe?

Edit: maybe I know what happens: when MQTTClient.publish calls MQTT_base.publish to republish the dup field is set to 0 and mosquitto probably barfs at that if it has indeed seen the packet before. IIUC dup needs to be set to 0 even if republishing on a fresh connection.

peterhinch commented 4 years ago

I think this is why further complexity is required to achieve qos==2 and why of necessity dupes occur if qos < 2. In practice discarding dupes is easy at application level.

tve commented 4 years ago

My comment wasn't about qos=2 nor was it about dupes. It was about a bug that caused you to incorrectly conclude that you had to retransmit using a new PID.

peterhinch commented 4 years ago

Describing this as a bug is overstating the case. A bug would be a program failure or behaviour non-compliant with the MQTT spec. In this instance a deliberate design decision was made which arguably was not optimal. I don't believe it's non-compliant, but if you can point me to a specific para in the spec I will reconsider.

Duplicate messages are a declared hazard of qos==1.

If these are problematic, an application subscribing at qos==1 to a qos==1 publication will implement de-duping, if only because the broker can legally send them. Application-level de-duping will use an application-level message ID and won't care about the PID.

The worst consequence of this behaviour is a small increase in network traffic and subscriber work, and then only in the uncommon instance of an outage experienced by the publisher.

I will review this in due course in conjunction with the MQTT spec, but I don't regard it as a priority.

tve commented 4 years ago

In practice discarding dupes is easy at application level.

@peterhinch You wrote the above. Can you elaborate for situations where messages are not idempotent?

peterhinch commented 4 years ago

The publisher embeds a unique message ID in the message. Each subscriber keeps a log of received ID's and discard messages whose ID is in that log. If either the MQTT library or the broker cause a duplicate message to be published, it will be discarded.

The application need not know or care about PID's: a PID is just a unique message ID. Producing one at application level is trivial.

Under the constraints of qos==1 the broker may send a duplicate to subscribers. Unless qos==2 can be used the application must solve the problem of non-idempotent messages whatever MQTT module is in use. I therefore don't see this as a meaningful issue with this module.