njouanin / hbmqtt

MQTT client/broker using Python asynchronous I/O
MIT License
799 stars 188 forks source link

Request for explanation around 3.4 syntax for @async.coroutine #163

Open bh4r4th opened 5 years ago

bh4r4th commented 5 years ago

While working with similar implementation to that of hbmqtt.plugin.logging plugin, I came across a commit for syntax of coroutines reverted back to 3.4 style using @asyncio.coroutine by @njouanin . What is the difference in both. Does it have any limitations with python 3.6.* implementations. Like: aync def on_mqtt_message_sent():.

Does ever on_mqtt_message_sent become unreachable with the newer syntax?

Link to the commit: https://github.com/beerfactory/hbmqtt/commit/4dcf8eb4776c7c6574d2a6120b6dbc8b8832df13

bh4r4th commented 5 years ago

Reason I asked because: I created a plugin similar to logger.py. In which I created a buffer which monitors PUBACK and PublishPacket and stores the file_name of json and packet_id. On successful PUBACK I deleted the file stored in my SBC.

This worked fine when I run on my Mac and 5 other SBC's. But failed to acknowledge properly on couple devices.

On SBC's I spin up as docker container and all the devices have same configuration and same version of dependencies. I only see a log for write_frame. There is no logs for PublishPacket but received PubAck logs.

My suspicion:

Any help/advice/suggestion on this issue type is much appreciated!

bh4r4th commented 5 years ago

My json file size is 1.7mb that I used. I think mqtt accepts 256mb of payload. Ref: http://www.steves-internet-guide.com/mqtt-protocol-messages-overview/

bh4r4th commented 5 years ago

Progress: I cloned the hbmqtt v0.9.5 locally into my project and added logs to pin-point the issue. So far found the internet upload speed on the box is 0.9MB/sec where as on other boxes it is around 5MB/sec. And In the _send_packet() of handler.py yield from packet.to_stream(self.writer) not yielded quick enough compared with other box.

bh4r4th commented 5 years ago

Update: PublishPacket build is inheriting MQTTPacket. Which is using WriterAdapter; Indeed looks like not an async write method.

For some reason if my internet speed is less than the packet size then, packet.to_stream() is not getting yielded instantly and making the code blocked for ever. No logs or exception handled in this space.

Looks like a bug. Temporary Fix: I split my json object into multiple chunks that are less than the internet speed and sent weach packet with a delay of 15 seconds.

Permanent fix(Not Tried): Change the WriterAdapter to StreamWriterAdapter which supports asyncio.streamWriter.

bh4r4th commented 5 years ago

Note: Even switching to QOS_0 or QOS_2 we cannot bypass this issue as both flows using the same _send_packet() method.

smurfix commented 5 years ago

No, there shouldn't be any difference between 3.4 and 3.5+ syntax. Are you able to share code to reproduce the problem?