Memory allocation fails after returning from imported module

[endico]

I invoke a calibration mode with __import__("calibrate") at the beginning of code.py

In calibrate.py, I set text in 2 labels, run some timers and press some buttons. It doesn't seem like this should use much memory. I noticed that I ran out of memory once after returning to code.py so I decided to save some memory by pre-loading glyphs. Now whenver I pre-load the glyphs I consistently crash in the original module while loading a bitmap. Without pre-loading I think the allocation error happened only once. font.load_glyphs(b'RPCabcdefghiklmnoqrstuwy1234567890,')

This is very odd because pre-loading glyphs should be saving memory, not using more. But mainly its odd because exiting the imported module should release its memory, right? Should I be calling the sub-module some other way?

Traceback (most recent call last):
  File "code.py", line 41, in <module>
  File "adafruit_imageload/__init__.py", line 57, in load
  File "adafruit_imageload/__init__.py", line 48, in load
  File "adafruit_imageload/bmp/__init__.py", line 59, in load
  File "adafruit_imageload/bmp/indexed.py", line 70, in load
MemoryError: memory allocation failed, allocating 28800 bytes

[kmatch98]

You can try using the bitmap_label.Label Library to save a little more RAM. You actually can initiate bitmap_label with an empty string if you want to. Something seems strange but I can’t quite guess what it is. Give it a try and let me know.

One more thing, I don’t think preloading glyphs will save memory. I think it just saves a little time. It’s not as big a difference anymore with the improve font loading of PCF files though.

[endico]

I have two source files and it turns out that only one of them was using bitmap_label.

Originally, code.py was using regular Label and pre-loading glyphs. calibrate.py used bitmap_label and caused a crash in code.py when that file tried to pre-load glyphs.

Switching code.py to bitmap_label didn't help and now I'm unable to use it in either file without a memory error after returning from configuration. (even if neither file preloads glyphs) Everything works fine if I don't try to calibrate first.

This is the right way to use bitmap_label, right? from adafruit_display_text import bitmap_label as label

I don't know if it matters, but I'm using a pcf font.

[jposada202020]

@endico The way you import label is fine. If later you use it as label Do yo have the font file and the code in your repo? maybe we could try it. Also what version of

Adafruit_CircuitPython_Bitmap_Font

are you using?. Thanks

[endico]

I'm using libraries from adafruit-circuitpython-bundle-6.x-mpy-20210321 and Adafruit CircuitPython 6.1.0 on 2021-01-21; Adafruit CLUE nRF52840 Express with nRF52840

Everything is checked in to this repository and is up to date as of 5 hours ago. That version was able to load glyphs in code.py but not in calibrate.py. To trigger the error you need to go into calibration mode by pressing both buttons on the Clue as its booting up. To speed things up you'll want to change the wait_timer on line 70 or calibrate.py to 1 or 2. I'll post a diff between the git version and what's currently on my Clue.

[endico]

$ diff code.py /Volumes/CLUE/code.py 
12c12,14
< from adafruit_display_text import label
---
> # from adafruit_display_text import label
> from adafruit_display_text import bitmap_label as label
> 
22,23d23
< blah = "BLAH"
< 
27d26
< print("blah=", blah)
40d38
< print("blah=", blah)

$ diff calibrate.py /Volumes/CLUE/calibrate.py 
8a9,10
> # from adafruit_display_text import label
> 
12,13d13
< blah = "BLAH BLAH"
< 
21c21
<     text=" ",
---
>     text="                                                                                             ",
29c29
<     text="",
---
>     text="             ",
70c70,71
<         wait_timer = 180 # wait at least 2 minutes for air to clear
---
>         wait_timer = 2 # wait at least 2 minutes for air to clear
> #        wait_timer = 180 # wait at least 2 minutes for air to clear
81c82
< print("done calibrating")
---
> print("done calibrating")
\ No newline at end of file

[kmatch98]

Here's a few more things. Minimize your imports to only what you are using.

For example, the clue library imports a lot of things. If you're not using all those sensors, then don't import them. Make your own pared-down Clue library with just the imports that you need.

If you're only using part of a library, you can only import selected functions like: from library import function

Also, if you're finished with a variable instance, you can call del variable. Then do a gc.collect() to call the memory garbage collector. If the variable isn't referenced by anything else it may get garbage collected.

Also, after you change your label text, try immediately calling a gc.collect(). Calling the garbage collector more often may help reduce your memory defragmentation a little.

To see where you're using RAM, do an import gc and then sprinkle print("1 memory free: {}".format(gc.mem_free())) statements figure out what's using it.

One other complication is that gc.mem_free() will tell you how much RAM is available, but it won't show you the distribution. Meaning, that if your memory is fragmented, there may not be a large contiguous chunk that you may need to have available. CircuitPython doesn't do any memory de-fragmentation to clear out a large space for you.

Anyway, that's a few ideas to try to debug your situation. I can't find where I put my NRF52840 board, so I can't verify anything for you on hardware. Maybe you can make sense out of some of these ideas and get something working.

[endico]

Dang, I was hoping that garbage collecting after the import returned would fix it, but no.

    __import__("calibrate")
    gc.collect()

[endico]

I'm already doing from adafruit_clue import clue Is it possible to narrow it down more? Stuff like the following doesn't work

from adafruit_clue import button_a
from adafruit_clue import clue.button_a

[kmatch98]

I would just hack the clue library and delete everything you don’t need.

On Mar 24, 2021, at 11:59 PM, endico @.***> wrote:

 I'm already doing from adafruit_clue import clue Is it possible to narrow it down more? Stuff like the following doesn't work

from adafruit_clue import button_a from adafruit_clue import clue.button_a — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[FoamyGuy]

Alternatively to using a modified CLUE library you could remove it all together and use core APIs to do the things you need. It should be possible to do everything without the helper library, I think it may save some RAM compared to using the helper.

[endico]

"hack the clue library and delete everything you don’t need."

I have no idea what you mean by this. Nothing I tried worked. I can import clue, but can't load any of its components individually and I can't delete bits after loading the entire clue class. But it doesn't matter because clue hardly uses any memory anyway.

Printing out the amount of free memory worked fine but its not clear why its running out because there seems to be plenty. The lowest I ever got was 33K. Maybe there's fragmentation but my labels aren't that large and there's lots of free memory so I don't see how it could get that fragmented. None of the labels are more than 80 characters and there are less than 10 labels. I never went below 67K available in calibrate.py. The image I'm trying to load in code.py is 30K.

One problem I do see is that hardly anything from calibrate.py is getting freed. Memory free at the last line of calibrate.py is almost the same as the line in code.py after calibrate.py returns. Most of the ram in calibrate.py is used by the senor object, the font, group and the text labels. I tried deleting those at the end of the module and garbage collecting but that made no difference.

Here's a run where I don't calibrate and the image loads successfully code.py 1: memory free: 149376 code.py 2: memory free: 82976 code.py 4: memory free: 82896 code.py 5 before imageload: memory free: 68768 code.py 6 after imageload: memory free: 33504

Here's one where I calibrate and the memory allocation fails on imageloading the bitmap. code.py 1: memory free: 149376 code.py 2: memory free: 82960 calibrate.py: Beginning of file memory free: 80368 calibrate.py- before loading clue: 1 memory free: 80192 calibrate.py- after loading clue: 2 memory free: 80048 calibrate.py: 3 memory free: 79744 calibrate.py: 4 memory free: 72368 calibrate.py: 5 memory free: 67344 calibrate.py: 6 memory free: 63632 done calibrating calibrate.py: 6 memory free: 64768 code.py 3: memory free: 67168 code.py 4: memory free: 67232 code.py 5 before imageload: memory free: 59104 Traceback (most recent call last): File "code.py", line 52, in File "adafruit_imageload/init.py", line 57, in load File "adafruit_imageload/init.py", line 48, in load File "adafruit_imageload/bmp/init.py", line 59, in load File "adafruit_imageload/bmp/indexed.py", line 70, in load MemoryError: memory allocation failed, allocating 28800 bytes

[endico]

So that's about 13K that got consumed by calibrate.py but wasn't given back after it returned. I'm calling in code.py with __import__("calibrate"). Should I be doing something else?

[kmatch98]

Sorry I was unclear using about saying “hack the clue library”, hope I can clear it up a bit.

I tried running your code on my PyPortal and it kept giving errors because I didn’t have all the libraries that are imported by the adafruit_clue library. I suspect that even though you are only importing the clue Class, it still probably imports all the files that are imported in the adafruit_clue library. Since you only need a few bits from the clue library it might reduce your RAM usage by copying the adafruit_clue file into a new file and deleting all the unnecessary imports and other code. That may give you a bit more RAM after all your imports are complete. But looking at your numbers, doesn't RAM is getting used there, so maybe I'm wrong. Or perhaps it could be causing some fragmentation. Anyway, if you don't need these imports then I'd get rid of it from your custom "hacked" version of the adafruit_clue library.

As you say your gc.mem_free shows you have sufficient RAM but you cannot allocate enough bytes for your images. This suggests that your memory is fragmented when it tries to allocate the image. Perhaps try importing anything that is large in memory, like your image or fonts, as early as you can in the code (before the RAM gets too bad fragmented).

As for freeing memory due to your calibrate function, there appear to be opportunities there. In the calibrate code, you create group and show it on the display. After you are done with this group, and the two items in it (text_label and timer_label), you need to remove any references to these variables. By "removing any references", you need to be sure that nothing in your code is still "using" these variables or what's in them. There are a couple ways I know of to "de-reference" these items.

1. Remove the two labels from group. You can do group.pop() twice.
2. Then, you can dereference the labels like this: a. del text_label and del timer_label to tell the gc that you don't think you need these anymore. b. Set text_label = None and timer_label=None
3. Before you de-reference group you should remove it from the display, so do display.show(None).
4. Now de-reference group by using del group or group=None.
5. Now run gc.collect() and print(gc.mem_free()) and see if the garbage collector can confirm that these variables no longer are referenced by anything and free the RAM they were using.

You can do del variable on any variables you won't need in the rest of the code, but probably the biggest RAM users are the two labels.

Anyway, you may find better ways of making this work but it's just a few things that might be able to get you closer to where you want to be. I suspect dealing with these things is a pain, but other options are to find something with more RAM. But I suspect if you do a little tweaking you can make it work.

As for the __import__("calibrate") I've never used __import__ like that, I usually just say import calibrate but I don't suspect that will matter from RAM usage. I read that using __import__ like that is a way that you can use a string for the input (so you can decide programmatically what files you want to import rather than hardcoding it in your file). I don't think you need that.

[endico]

I just noticed that switching from bitmap_label to label avoids a crash. bitmap_label is saving about 1k and apparently causing fragmentation. I don't really want to switch to this though since its noticeably slower at drawing the text.

[kmatch98]

Here's one more crack at it by defining calibrate as a function, see the def calibrate(). I think this is really what you are trying to achieve, but perhaps you have a specific need for calibrate as a separate file?

I ran this on the pyportal. I had to remove some of your clue-specific statements so it would run. From my output, I am able to recover practically all the memory after repeatedly running calibrate:

Here is my output:

code.py output:
0 gc.mem_free: 197216
1 gc.mem_free: 189952
2 gc.mem_free: 189952
3 gc.mem_free: 189952
4 gc.mem_free: 189952
5 gc.mem_free: 189952
6 gc.mem_free: 189952

Here is my simplified code

import time
import gc
import board
import displayio
import adafruit_imageload
from adafruit_bitmap_font import bitmap_font
from adafruit_display_text import bitmap_label as label
#from adafruit_clue import clue
import adafruit_scd30
from digitalio import DigitalInOut, Direction, Pull

def calibrate():
    font = bitmap_font.load_font("/fonts/LCD14-64-Mod5.pcf")
    # font.load_glyphs(b'RPCabcdefghiklmnoqrstuwy1234567890,')

    color = 0xFFFFFF

    text_label = label.Label(
        font,
        text=" ",
        color=color,
        line_spacing=1.1,
        x=10,
        y=30,
        )
    timer_label = label.Label(
        font,
        text="",
        color=color,
        line_spacing=1.0,
        anchor_point=(0.5, 0.0),
        anchored_position=(display.width / 2, 160),
        )
    group = displayio.Group(max_size=4)
    group.append(text_label)
    group.append(timer_label)

    display.show(group)

    initiation_timer = 10
    while initiation_timer > 0:
        initiation_timer = initiation_timer - 1
        if False: #(clue.button_a or clue.button_b) :
            text_label.text = "Release\nbuttons to\nbegin calibration"
            timer_label.text = str(initiation_timer)
        else:
            # while (clue.button_a or clue.button_b):
            #     pass
            timer_label.text = " "
            text_label.text = "Press button\nwhen you are\noutdoors"
            # while not (clue.button_a or clue.button_b):
            #     pass
            # while (clue.button_a or clue.button_b):
            #     pass
            text_label.text = "Press button\nwhen sensor is\nin the shade"
            # while not (clue.button_a or clue.button_b):
            #     pass
            # while (clue.button_a or clue.button_b):
            #     pass
            text_label.text = "Press button\nwhen sensor is\nprotected from\nthe wind"
            # while not (clue.button_a or clue.button_b):
            #     pass
            # while (clue.button_a or clue.button_b):
            #     pass
            text_label.text = "Press button,\nput down the\nsensor and stand\nback during\ncalibration"
            # while not (clue.button_a or clue.button_b):
            #     pass
            text_label.text = "Calibrating"
            timer_label.text = " "
            wait_timer = 180 # wait at least 2 minutes for air to clear
            while wait_timer > 0:
                timer_label.text = str(wait_timer)
                wait_timer = wait_timer - 1
                time.sleep(0.01)
            timer_label.text = "0"

            #scd30.forced_recalibration_reference = 400
            break
        time.sleep(0.05) # sleep while waiting to verify this wasn't accidental
    timer_label.text = "0"
    #print("done calibrating")

    display.show(None)
    group.pop()
    group.pop()
    timer_label=None
    text_label=None
    group = None
    font = None
    del timer_label
    del text_label
    del group
    del font

##############
#  Here is where the main code starts
##############

# --| User Config |----
CO2_CUTOFFS = (1000, 2000, 5000)
UPDATE_RATE = 2
# ---------------------

#blah = "BLAH"

display=board.DISPLAY

count=0

while True:

    gc.collect()
    print("{} gc.mem_free: {}".format(count, gc.mem_free()))
    calibrate()

    count+=1

#scd30 = adafruit_scd30.SCD30(board.I2C())

[endico]

Thanks! After talking to people in discord I added supervisor.reload() to the end of calibrate.py and it fixed my problem. I forget now why I put it in a separate file. Maybe you're right and putting it in a function is better. I'll try that too.

[kmatch98]

Just remembered. You really don’t need all the del commands at the end. Once the calibrate function is over all the local variables can get collected.

[endico]

Ok, I just checked in a couple more changes to clean this up. (commit 937facf )

1. Load the bitmap image first thing after the imports to make sure it finds a contiguous block of memory.
2. Got rid of calibrate.py and put that code into a function.

I still noticed that there's still a missing 16k of memory after returning from the function call but later on after creating the new font, group and labels, free mem goes up to 31k which is the same as I get on runs where I don't calibrate. I guess those variables are still referenced by the board and aren't getting freed until later when they're replaced.

This commit contains the debug code in case I want to look at it again.

code.py output: gc.mem_free: after load gc 147904 gc.mem_free: before bitmap 81536 gc.mem_free: after bitmap 46512 gc.mem_free: after scd30 load 46336 gc.mem_free: before calibrate 46320 gc.mem_free: after calibrate 30144 gc.mem_free: after fonts 24832 gc.mem_free: before update_display loop 31008

[kmatch98]

Perhaps the garbage collector hasn’t run after you leave the calibrate function. You can force it to run by adding a gc.collect() before your print gc.mem_free().

Or to verify, run a print gc.mem_free() then gc.collect() and another print gc.mem_free().

Glad to see you’ve made good progress!

[endico]

Yeah, I added a collect before each print statement. 🤷‍♀️