search

adafruit / circuitpython

CircuitPython - a Python implementation for teaching coding with microcontrollers
https://circuitpython.org
Other
4.04k stars 1.19k forks source link

Memory error after not so much activity: fragmentation, or worse? #2885

Open dhalbert opened 4 years ago

dhalbert commented 4 years ago

from @kevinjwalters: https://forums.adafruit.com/viewtopic.php?f=60&t=165359 (this example from second post in that thread)

Adafruit CircuitPython 5.3.0 on 2020-04-29; Adafruit CLUE nRF52840 Express with nRF52840
>>> from adafruit_clue import clue
>>> import time, gc, random
>>> gc.collect() ; gc.mem_free()
89872
>>> a=[(1.0, 2.0, 3.0) for x in range(500)
... ]
>>>
>>> def read(n):
...     for idx in range(n):
...         a[idx] = clue.magnetic
...
...
...
>>> gc.collect() ; gc.mem_free()
71472
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71488
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> t1 = time.monotonic_ns() ; read(500) ; t2 = time.monotonic_ns()
>>> print((t1-t2)/1e9)
MemoryError: memory allocation failed, allocating 512 bytes
dhalbert commented 4 years ago

@kevinjwalters were you using the .py or the .mpy version of the CLUE library?

kevinjwalters commented 4 years ago

.mpy and a slightly old one - it's from adafruit-circuitpython-bundle-5.x-mpy-20200327

kevinjwalters commented 4 years ago

This is telling, it's something to do with the evaluation of the result in the argument to a function or specifically to print.

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.3.0 on 2020-04-29; Adafruit CLUE nRF52840 Express with nRF52840
>>> from adafruit_clue import clue
>>> import time, gc, random
>>> gc.collect() ; gc.mem_free()
89872
>>> a=[(1.0, 2.0, 3.0) for x in range(500)
... ]
>>> gc.collect() ; gc.mem_free()
71648
>>> read(500)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'read' is not defined
>>> def read(n):
...     for idx in range(n):
...         a[idx] = clue.magnetic
...
...
...
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> read(500)
>>> gc.collect() ; gc.mem_free()
71504
>>> t1 = time.monotonic_ns() ; read(500) ; t2 = time.monotonic_ns()
>>> t1-t2
-1782595000
>>> (t1-t2)/1e9
-1.78259
>>> res = (t1-t2)/1e9
>>> print(res)
-1.78259
>>> print((t1-t2)/1e9)
MemoryError: memory allocation failed, allocating 512 bytes
kevinjwalters commented 4 years ago

It's got weirder, following on from commands in previous comment:


>>> t1-t2
-1782595000
>>> print(-1782595000/1e9)
-1.78259
>>> print(t1/1e9)
18424.0
>>> print((t1-t2)/1e9)
MemoryError: memory allocation failed, allocating 512 bytes
>>> print((t1+t2)/1e9)
MemoryError: memory allocation failed, allocating 512 bytes
>>> print(t1/1e9 - t2/1e9)
-1.78125
>>> print((t1-t2)/1e9)
MemoryError: memory allocation failed, allocating 512 bytes
kevinjwalters commented 4 years ago

There's something about brackets that it doesn't like

Adafruit CircuitPython 5.3.0 on 2020-04-29; Adafruit CLUE nRF52840 Express with nRF52840
>>>
>>>
>>> from adafruit_clue import clue
>>> import time, gc, random
>>> gc.collect() ; gc.mem_free()
91824
>>> a=[(1.0, 2.0, 3.0) for x in range(500)]
>>> gc.collect() ; gc.mem_free()
73616
>>> def read(n):
...     for idx in range(n):
...         a[idx] = clue.magnetic
...
...
...
>>> t1 = time.monotonic_ns() ; read(500) ; t2 = time.monotonic_ns()
>>> gc.collect() ; gc.mem_free()
73024
>>> print((t1-t2)/1e9)
-1.8214
>>> t1 = time.monotonic_ns() ; read(500) ; t2 = time.monotonic_ns()
>>> print((t1-t2)/1e9)
MemoryError: memory allocation failed, allocating 512 bytes
>>> gc.collect() ; gc.mem_free()
72992
>>> print((t1-t2)/1e9)
MemoryError: memory allocation failed, allocating 512 bytes
>>> var1=t1+t2
>>> var2=(t1+t2)
>>> var3=((t1+t2))
MemoryError: memory allocation failed, allocating 512 bytes
>>> var4=(1,2,3,4,5,6,7,8,10,11,12,13,14)
>>> var5=(t1, t2)
>>> var6=((t1+t2))
MemoryError: memory allocation failed, allocating 512 bytes
dhalbert commented 10 months ago

Retested with 8.2.7 and latest libraries. It fails even earlier:

Adafruit CircuitPython 8.2.7 on 2023-10-19; Adafruit CLUE nRF52840 Express with nRF52840
>>> from adafruit_clue import clue
>>> import time, gc, random
>>> gc.collect() ; gc.mem_free()
87200
>>> a=[(1.0, 2.0, 3.0) for x in range(500)
... ]
>>> def read(n):
...     for idx in range(n):
...         a[idx] = clue.magnetic
...         
...         
... 
>>> 
>>> 
>>> gc.collect() ; gc.mem_free()
68800
>>> read(500)
>>> gc.collect() ; gc.mem_free()
68832
>>> read(500)
>>> gc.collect() ; gc.mem_free()
MemoryError: memory allocation failed, allocating 384 bytes
>>> read(500)
MemoryError: memory allocation failed, allocating 384 bytes
tannewt commented 10 months ago

I've debugged this. It is a pathological case where the .magnetic code takes a couple allocations and then creates the tuple that is stored in a. This leaves short gaps between each tuple that is held for longer.

Below is the output of gc_dump_alloc_table() for the last memory pool when the memory allocation fails. Each character is a block. T is tuple, = continues the object and . is free. It is severely fragmented.

I think this only happens from the REPL because the compilation process that runs the input has larger allocations than can fit. When the code is in a file, that cost is paid up front before this code is run.

GC memory layout; from 0x20028760:
00000000: ..............T=..................T=..................T=........
00000400: ..........T=..................T=...............T=..........T=...
00000800: .......T=.............T=.........T=..........T=..........T=.....
00000c00: ........T=.........T=..........T=..........T=.............T=....
00001000: .....T=..........T=..........T=.............T=.........T=.......
00001400: ...T=..........T=.............T=.........T=..........T=.........
00001800: .T=.............T=.........T=..........T=..........T=...........
00001c00: ..T=.........T=..........T=..........T=.............T=.........T
00002000: =..........T=..........T=.............T=.........T=..........T=.
00002400: .........T=.............T=.........T=..........T=..........T=...
00002800: ..........T=.........T=.......h....h=.h==h..T=.........T=T=.....
00002c00: .....T=..T=......T=T=........T=T=......T=..T=....T=....T=..T=...
00003000: ...T=T=........T=T=......T=..T=....T=....T=..T=......T=T=.......
00003400: .T=T=......T=..T=....T=....T=..T=......T=T=........T=T=......T=.
00003800: .T=....T=....T=..T=......T=T=........T=T=......T=..T=....T=....T
00003c00: =..T=......T=T=........T=T=......T=..T=....T=....T=..T=......T=T
00004000: =........T=T=......T=..T=....T=....T=..T=......T=T=........T=T=.
00004400: .....T=..T=....T=....T=..T=......T=T=........T=T=h=hh=hT=h======
00004800: =T=h=......T=........T=........T=........T=........T=........T=.
00004c00: .......T=........T=........T=........T=........T=........T=.....
00005000: ...T=........T=........T=........T=........T=........T=........T
00005400: =........T=........T=........T=........T=........T=........T=...
00005800: .....T=........T=........T=........T=........T=........T=.......
00005c00: .T=........T=........T=........T=........T=........T=........T=.
00006000: .......T=........T=........T=........T=........T=........T=.....
00006400: ...T=........T=........T=........T=........T=........T=........T
00006800: =........T=........T=........T=........T=........T=........T=...
00006c00: .....T=........T=........T=........T=........T=........T=.......
00007000: .T=........T=........T=........T=........T=........T=........T=.
00007400: .......T=........T=........T=........T=........T=........T=.....
00007800: ...T=........T=........T=........T=........T=........T=........T
00007c00: =........T=........T=........T=........T=........T=........T=...
00008000: .....T=........T=........T=........T=........T=........T=.......
00008400: .T=........T=........T=........T=........T=........T=........T=.
00008800: .......T=........T=........T=........T=........T=........T=.....
00008c00: ...T=........T=........T=........T=........T=........T=........T
00009000: =........T=........T=........T=........T=........T=........T=...
00009400: .....T=........T=........T=........T=........T=........T=.......
00009800: .T=........T=........T=........T=........T=........T=........T=.
00009c00: .......T=........T=........T=........T=........T=........T=.....
0000a000: ...T=........T=........T=........T=........T=........T=........T
0000a400: =........T=........T=........T=........T=........T=........T=...
0000a800: .....T=........T=........T=........T=........T=........T=.......
0000ac00: .T=........T=........T=........T=........T=........T=........T=.
0000b000: .......T=........T=........T=........T=........T=........T=.....
0000b400: ...T=........T=........T=........T=........T=........T=........T
0000b800: =........T=........T=........T=........T=........T=........T=...
0000bc00: .....T=........T=........T=........T=........T=........T=.......
0000c000: .T=........T=........T=........T=........T=........T=........T=.
0000c400: .......T=........T=........T=........T=........T=........T=.....
0000c800: ...T=........T=........T=........T=........T=........T=........T
0000cc00: =........T=........T=........T=........T=........T=........T=...
0000d000: .....T=........T=........T=........T=........T=........T=.......
0000d400: .T=........T=........T=........T=........T=........T=........T=.
0000d800: .......T=........T=........T=........T=........T=........T=.....
0000dc00: ...T=........T=........T=........T=........T=........T=........T
0000e000: =........T=........T=........T=........T=........T=........T=...
0000e400: .....T=........T=........T=........T=........T=........T=.......
0000e800: .T=........T=........T=........T=........T=........T=........T=.
0000ec00: .......T=........T=........T=........T=........T=........T=.....
0000f000: ...T=........T=........T=........T=........T=........T=........T
0000f400: =........T=........T=........T=........T=........T=........T=...
0000f800: .....T=........T=........T=.
kevinjwalters commented 10 months ago

Thanks for looking into that.

I think this started because I wanted to see what rate I could read at and how close I could get to the sensor's maximum rate. I think there are some other recent efforts afoot for bulk sampling or continuous sampling of sensors perhaps via DMA where the hardware permits? Had to look this up, I'm thinking of analogbufio. Is there anything afoot for something similar with an interrupt driving approach for i2c with nRF52 for my scenario?

tannewt commented 10 months ago

Is there anything afoot for something similar with an interrupt driving approach for i2c with nRF52 for my scenario?

Not yet. But I could see us adding something like that to "read X register N times over i2c".