MQTT outbox dequeuing mechanism for esp_mqtt_client_enqueue (IDFGH-13400)

[Teesmo]

How does esp_mqtt_client_enqueue dequeue? Is it a FIFO mechanism or a LIFO (or hybrid), I have noticed instances where the most recent data in the queue is published first, or is in conjuction with the (FIFO) queue(i.e temporarily interrupts FIFO dequeuing).

[euripedesrocha]

Hi @Teesmo we always push items to the end. When trying to publish, the first valid occurrence is selected. For recently enqueued this translates to a FIFO, if we are trying to re transmit messages the one selected is the first that had expired the timeout for retransmission.

So it is always FIFO, but with some different criteria depending on scenario.

Is this behavior affecting your system in some sense?

[Teesmo]

I need a strictly FIFO pipeline because I need data transferred in the order it was extracted from sensors (time-series is crucial here). I need this data not only for time-series but also in order processing, as I have external systems that interact with this system, and can alter the behaviour of this system. If I understand correctly, the fisrt valid occurrence should be the one that had expired the retransmission timeout. Essentially, this somehow 'guarantees' in order (or FIFO)? If that's the case, then I think it could work for me. All I need is the FIFO guarantee

[euripedesrocha]

@Teesmo could you add more information like: QoS of messages, are you using publish or enqueue function?

[Teesmo]

I am using the enqueue function, QoS 1

[euripedesrocha]

@Teesmo thanks for clarification. On the way the outbox is managed I believe that you will have the messages sent in the order that they are pushed to the outbox. You may set the MQTT_MSG_ID_INCREMENTAL to track the order of the messages created. Can you share how you are detecting the order of messages sent?

If I understand correctly, the fisrt valid occurrence should be the one that had expired the retransmission timeout. Essentially, this somehow 'guarantees' in order (or FIFO)?

From the mqtt client we are publishing in the order that the messages are created, if the broker didn't answer with the control message before the timeout we retry. So let's assume that you have 2 messages enqueed and one that have expired the timeout. We'll send the first of the 2 published and the one with the timeout expired. If the broker missed the one that we are retransmiting it might be that the message we sent before is received first. Over that we have no control and you should use other mechanisms in you system, like ordered message ID as I suggested or a time stamp in the message.