search

peterhinch / micropython-mqtt

A 'resilient' asynchronous MQTT driver. Recovers from WiFi and broker outages.
MIT License
549 stars 116 forks source link

Memory allocation error reading new message #83

Closed TheBodger closed 1 year ago

TheBodger commented 1 year ago

Certain new messages are causing a memory error as reported below. The error seems to point to the _as_read function, maybe when the code allocates the buffer or data for a new message, i assume with the aync nature of the code meaning there can be multiple message instances at the same time momentarily overwhelming the memory capacity of the pico. Would it be possible within the async nature of the code to add a check before the allocation to ensure there is enough memory before trying to allocate and what would be the best way of handling a lack of memory without mucking up the whole process ? txs for a great app

RAM free 125104 alloc 40976 Topic: "MM/metofficecot1/MMQTRelay" MessageLen: "6511" Retained: False Topic: "MM/RSSFeed2/MMQTRelay" MessageLen: "15067" Retained: False Topic: "MM/metofficecot/MMQTRelay" MessageLen: "6510" Retained: False Topic: "MM/RSSFeed2/MMQTRelay" MessageLen: "2944" Retained: False Topic: "MM/RSSFeed2/MMQTRelay" MessageLen: "1299" Retained: False Topic: "MM/MMMTwitter/MMQTRelay" MessageLen: "13841" Retained: False Topic: "MM/MMFP5/MMQTRelay" MessageLen: "6413" Retained: False Topic: "MM/MMMTwitter/MMQTRelay" MessageLen: "11205" Retained: False Task exception wasn't retrieved future: coro= <generator object '_handle_msg' at 20015220> Traceback (most recent call last): File "uasyncio/core.py", line 1, in run_until_complete File "/lib/mqtt_as.py", line 581, in _handle_msg File "/lib/mqtt_as.py", line 466, in wait_msg File "/lib/mqtt_as.py", line 171, in _as_read MemoryError: memory allocation failed, allocating 33268 bytes RAM free 125296 alloc 40784

peterhinch commented 1 year ago

MemoryError: memory allocation failed, allocating 33268 bytes

This seems to imply you are receiving very long messages. Is this the case?

There was a recent change that may have affected the ability to receive long messages. I have pushed an update reverting this change. Please could you re-test and report your results.

TheBodger commented 1 year ago

The message sizes are reported next to the topic names, I am not sure if 20k would be considered large (possibly). I took a copy of the new module and used that. As you can see on 2 consecutive runs the error repeats but on the first very quickly, the second runs a bit more. Once the error is hit, the app continues without any activity and after a while it restarts the connection and msgs flow again .. until the next memory error. thanks

Checking WiFi integrity. Got reliable connection Connecting to broker. Connected to broker. client connected We are connected to broker. Subscribing to: home/garage/# Subscribing to: MM/# home/garage/NAS 55 home/garage/NASStatus 44 home/garage/NASTime 130 home/garage/NASMsg 96 Task exception wasn't retrieved future: coro= <generator object '_handle_msg' at 200146f0> Traceback (most recent call last): File "uasyncio/core.py", line 1, in run_until_complete File "/lib/mqtt_as.py", line 615, in _handle_msg File "/lib/mqtt_as.py", line 488, in wait_msg File "/lib/mqtt_as.py", line 203, in _as_read MemoryError: memory allocation failed, allocating 25137 bytes RAM free 125312 alloc 40768

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── MicroPython v1.19.1-88-g99c258977 on 2022-06-30; Raspberry Pi Pico W with RP2040 Type "help()" for more information.

%Run -c $EDITOR_CONTENT Checking WiFi integrity. Got reliable connection Connecting to broker. Connected to broker. client connected We are connected to broker. Subscribing to: home/garage/# Subscribing to: MM/# home/garage/NAS 55 home/garage/NASStatus 44 home/garage/NASTime 130 home/garage/NASMsg 96 MM/RSSFeed1/MMQTRelay 28508 MM/metofficeAscot/MMQTRelay 6034 MM/MMMTwitter/MMQTRelay 10155 MM/MMFP5/MMQTRelay 506 MM/RSSFeed2/MMQTRelay 2336 MM/metofficeAscot1/MMQTRelay 6035 MM/RSSFeed10/MMQTRelay 28509 home/garage/NAS 55 home/garage/NASStatus 44 home/garage/NASTime 130 home/garage/NASMsg 96 home/garage/NAS 55 home/garage/NASStatus 44 home/garage/NASTime 130 home/garage/NASMsg 96 MM/metofficeAscot1/MMQTRelay 6037 MM/metofficeAscot/MMQTRelay 6036 MM/MMFP5/MMQTRelay 6548 MM/MMMTwitter/MMQTRelay 14316 home/garage/NAS 55 MM/MMMTwitter/MMQTRelay 10086 home/garage/NASStatus 44 home/garage/NASTime 130 MM/MMMTwitter/MMQTRelay 751 RAM free 119536 alloc 46544 MM/MMMTwitter/MMQTRelay 12060 home/garage/NASMsg 96 Task exception wasn't retrieved future: coro= <generator object '_handle_msg' at 200146f0> Traceback (most recent call last): File "uasyncio/core.py", line 1, in run_until_complete File "/lib/mqtt_as.py", line 615, in _handle_msg File "/lib/mqtt_as.py", line 488, in wait_msg File "/lib/mqtt_as.py", line 203, in _as_read MemoryError: memory allocation failed, allocating 33460 bytes RAM free 119360 alloc 46720 RAM free 119360 alloc 46720

peterhinch commented 1 year ago

On further thought I realised that this version would not fix the problem and I will revert the change.

In principle the MQTT protocol is designed to handle huge messages. However there is a practical limit on a micocontroller running MicroPython. This is because whatever approach is used for the read, the message must be assembled into a single array of bytes. This requires a block of contiguous RAM. In practice available RAM becomes fragmented, so even with your 119360 bytes free it's extremely unlikely that you'll be able to allocate a buffer of 33460 bytes.

You need to use much smaller messages.

TheBodger commented 1 year ago

Hi

It is missing the _as_write method

neil

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: Peter @.> Sent: 26 July 2022 16:40 To: @.> Cc: @.>; @.> Subject: Re: [peterhinch/micropython-mqtt] Memory allocation error reading new message (Issue #83)

Closed #83https://github.com/peterhinch/micropython-mqtt/issues/83 as completed.

— Reply to this email directly, view it on GitHubhttps://github.com/peterhinch/micropython-mqtt/issues/83#event-7066088000, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJ4QC4F6IYAZVC4UYRADMZTVWABHBANCNFSM54VNKRAA. You are receiving this because you authored the thread.Message ID: @.***>

peterhinch commented 1 year ago

It is missing the _as_write method

Apologies. Now fixed.