How to use optimistic_acknowledgement?

[canique]

If you set optimistic_acknowledgement = False, how would you acknowledge received messages manually in the on_message handler?

[Mixser]

Hi @canique, if you switched off the optimistic acknowledgemen, then ack will be published only after your on_message callback will be processed. You don't need to send it manually.

To control which ack need to send your on_message callback must return one of PubRecReasonCode value;

def _handle_qos_2_publish_packet(self, mid, packet, print_topic, properties):
    if self._optimistic_acknowledgement:
        # send ack before on_message will be processed
        self._send_pubrec(mid)
        run_coroutine_or_function(self.on_message, self, print_topic, packet, 2, properties)
    else:
        # send ack right after on_message will be processed, and use its result as ack status code;
        run_coroutine_or_function(self.on_message, self, print_topic, packet, 2, properties,
                                      callback=partial(self.__handle_publish_callback, qos=2, mid=mid))

[canique]

thanks for the fast reply

[canique]

I have tried setting optimistic_acknowledgement=False but when set, my on_message() handler does not get called anymore.

I am returning return PubRecReasonCode.SUCCESS in the on_message handler. I have tried with QOS 1 and QOS 2.

But the handler does not get called (not even the first line) when I disable optimistic_acknowledgement. I only receive messages with retain=True at startup. Then no further msgs are coming in.

[Mixser]

Hi @canique

Please share your code to investigate your problem;

[canique]

When you say QoS 2 - do you mean the publish must be QoS 2, or is it enough if the subscribe is QoS 2? In my test scenario I am receiving QoS 1 messages, but when subscribing I've used QoS 2.

[Mixser]

When you say QoS 2 - do you mean the publish must be QoS 2, or is it enough if the subscribe is QoS 2?

No, sorry I made a mistake. You are right, optimistic_acknowledgement should work both QoS1 and QoS2.

[canique]

import asyncio
import os, sys, configparser
import signal
import time

from gmqtt import Client as MQTTClient
from gmqtt.mqtt.constants import PubRecReasonCode

import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

STOP = asyncio.Event()

def on_connect(client, flags, rc, properties):
    print('Connected')
    client.subscribe('+/sensors/#', qos=2)

def on_message(client, topic, payload, qos, properties):
    print('RECV MSG on topic {} QOS {}'.format(topic, qos))

    if properties['retain'] == 1:
        print('Skipping retain message')

    return PubRecReasonCode.SUCCESS

def on_disconnect(client, packet, exc=None):
    print('Disconnected')

def on_subscribe(client, mid, qos, properties):
    print('SUBSCRIBED')

def ask_exit(*args):
    STOP.set()

async def main(broker_host, username, password):

    client = MQTTClient("test3", clean_session=False, session_expiry_interval=3600, optimistic_acknowledgement=False)

    client.on_connect = on_connect
    client.on_message = on_message
    client.on_disconnect = on_disconnect
    client.on_subscribe = on_subscribe

    client.set_auth_credentials(username, password)

    await client.connect(broker_host, 1883, keepalive=60)

    await STOP.wait()

    await client.disconnect()

if __name__ == '__main__':
    loop = asyncio.get_event_loop()

    parser = configparser.RawConfigParser()
    host = ''
    user = ''
    password = ''

    with open(sys.argv[1]) as stream:
        parser.read_string(stream.read())
        host = parser['MQTT'].get('host', 'localhost')
        user = parser['MQTT'].get('username', '')
        password = parser['MQTT'].get('password', '')

    loop.add_signal_handler(signal.SIGINT, ask_exit)
    loop.add_signal_handler(signal.SIGTERM, ask_exit)

    loop.run_until_complete(main(host, user, password))

[canique]

The output of the above code is (I've edited the topic names):

Connected SUBSCRIBED RECV MSG on topic ... QOS 1 Skipping retain message RECV MSG on topic ... QOS 1 Skipping retain message RECV MSG on topic ... QOS 1 Skipping retain message RECV MSG on topic ... QOS 1 Skipping retain message

After that there should be incoming messages. But the handler does not get called...

[Mixser]

Thanks, I will test it with my env. Also, you can switch on DEBUG level of logger by adding this as early as possible:

import logging
logger = logging.getLogger('gmqtt')
logger.setLevel(logging.DEBUG)

And see, that you are really getting messages from the Broker, after all your retained messages has been successfully received

[canique]

When I add the logging code, nothing changes in console output. Do I need to change something else?

Yes I am getting messages, those that are retained. But not a single message that is not retained. I have a similiar code (with a different client ID of course), that does not change the optimistic_acknowledgement setting. It is getting all messages. If I only change that option, everything is working normal.

[canique]

As soon as I enable optimistic_ack I get all the messages that the broker could not send me in the meantime - all of them at once @ startup. So there must be something blocking those msgs from coming in. There are settings like number of in-flight msgs e.g. @ the broker. If let's say 10 msgs have been transmitted but not acked, then the broker won't send any more until it gets an ack. I suspect that somehow the code does not send this ACK, even though I return a success code.

[canique]

I'm using python 3.7 by the way.

[Mixser]

Ok, I see. At this moment I can't help you (until tomorrow); Please, try to replace PubRecReasonCode.SUCCESS with PubRecReasonCode.SUCCESS.value, may be this will help you;

[canique]

I added .value - unfortunately that did not fix it. I'll wait for tomorrow :)

[Mixser]

And another quick try to fix your problem - make your on_message callback async; As I see in the code - we have realized run_coroutine_or_function in incorrect way in case when on_message is sync;

[canique]

That did the trick! Changing "def on_message" to "async def on_message" made it work (while still using "return PubRecReasonCode.SUCCESS")

Thanks!

[Mixser]

🥳🥳🥳🥳

Great, thanks for your sample of code. We will fix this behavior in next release;

[canique]

Thanks a lot :+1:

[canique]

I have done some tests now with QoS 1 messages (both publish + subscribe).

Even if I send a reason code of 128 (unspecified error), I never get to see the message again. It is not re-sent. I disconnect the client, and on reconnect all the messages with pubacks with reason code 128 are not retransmitted.

I've added a log statement in handler.py: _send_puback(): The log output looks like sending puback with reason 128 mid 151 sending puback with reason 128 mid 251 sending puback with reason 128 mid 351

Any idea? I'm using community edition of HiveMQ broker...

[canique]

I ran a test with mosquitto broker now. Thr broker receives the PUBACK with the reason code correctly. The broker behaves similar to HiveMQ. Even if a PUBACK with reason code 128 is sent, the broker will not resend the message upon reconnect.

BUT: if no PUBACK is sent at all (by leaving out the return statement), the broker will resend all those unacked messages upon reconnect.

So a negative puback does not lead to a retranmission. But a missing puback does.

[Mixser]

Hi @canique, according to the specification this is a correct behavior

If PUBACK or PUBREC is received containing a Reason Code of 0x80 or greater the corresponding PUBLISH packet is treated as acknowledged, and MUST NOT be retransmitted [MQTT-4.4.0-2].