8 MB psram support

[nliaudat]

The Feature

AI-Tinker boards and all the esp-cam series have 8Mb of psram, but ESP32 can only map up to 4 MB PSRAM into its address space.

To access all 8 MB, the Espressif implements a "bank" mechanism called himem allocation API

The Himem subsystem does this by reserving some amount of address space, then allowing applications to swap in and out normally unreachable ranges of physical SPI RAM. While this does not allow transparent access in the way memory allocated with malloc() does, it does provide an usable way to store data for e.g., the captures images, buffers, tflites models, in the upper 4 MiB. That's like a storage space for memory objects

Moving large memory objets to himem space can free available RAM memory.

Sample code : https://github.com/espressif/esp-idf/blob/master/examples/system/himem/README.md

Dev env with sdkconfig is ready in https://github.com/jomjol/AI-on-the-edge-device/pull/1841

[caco3]

Once we start using the upper half of the 8 MB, we lose compatibility to all devices which have only 4 MB! Since (at least I) only have devices with 8 MB, it would be impossible to test that we did not break anything for those boards!

@Slider0007 added some great improvements to the RAM usage, so I do not see a reason now to break compatibility!

[nliaudat]

A simple check at boot will get the information and set a variable to use or not extra 4mb psram. Nothing will break compatibility

[caco3]

but what is the advantage if we still have to be able to work with 4 MB?

[nliaudat]

The main advantage can be moving all temporary things in sdcard to HIMEM. The overall performance will be increased as the access to sdcard is much slower than RAM. It' can preserve the sdcard life

Big memory things like tflite model can also be pushed in HIMEM at start and then read faster.

[caco3]

It' can preserve the sdcard life

That is only the case if you write to the SD-Card!

During a round, no images/data gets stored on the SD card. Exceptions:

• Data logging enabled
• Logging in Info or lower (but info only has some entries per minute)

[nliaudat]

Do we really know the amount of free RAM used in standby or in running a round ?

[caco3]

I don't, but @Slider0007 invested quite some time to investigate and improve it.

[Slider0007]

I invested quite some time to understand memory usage in relation of the different processing steps. Right now, we have a good picture how the memory usage looks like also during live time. To make it more transparant for everyone and also for diagnostic purpose I inlcuded a REST handler to call the available heap (internal RAM and external PSRAM, actual and lowest free memory since device start) at any time you want. Just call http:// IP /heap. (It's already included in rolling branch.)

output e.g. Heap Total: 1783966 | SPI Free: 1745815 | SPI Larg Block: 1736704 | SPI Min Free: 728151 | Int Free: 38151 | Int Larg Block: 30720 | Int Min Free: 13879

Some weeks ago I also played with himen memory a bit. With actual design goals of @jomjol to keep the SD card for portability reason (which is totally understandable) and last modification which reduced memory usage in my opinion right now the advantages to use himen are quite low. I agree that it would be possible to maybe get rid of /img_tmp files by storing in himen, but as long SD is still neccessary in my opinion it's not worth it. Also my analysis showed that with the latest changes we have worst case (using the lagest model, ca. 1,2MB, and switch on all available services) still ca. 600kB free PSRAM is available. Also for internal RAM we should on the safe side right now.

I also agree that we have to have an eye on RAM and stack heap usage whenever bigger changes or new features will be intergrated in the future. For tracking the stack sizes the flag TASK_ANALYSIS_ON can be activated to keep track of that with same REST handler heap. Running out of RAM or stack heap could cause unforseen issues which are difficult to track. Therefore this should be avoided at any cost.

The biggest disadvantage of himem is that use have to do our own memory management because of banking access (no malloc). The means it would be more difficult to use, maintain and check for memory leakages. Also a lot of users complain that they have to less or bad memory, so maybe it would be better not using the device with a possible feastures but make the device as robust as possible with less ressources as possible.

Long term it could be possible to think about using more RAM incl. himen to maybe get rid of SD card or implement cutting edge features but this would be change a lot and needs to be prepared well.

To sum it up in my humble and personal opinion: The advantage is right now to low in comparison to the risks of using himem (at least short term).

[nliaudat]

@Slider0007 I agree with you partially :

-RAM is always better than sdcards +HIMEM allocation is not well documented and more difficult to implement -Your heap output "Int Min Free: 13879" aka "heap_caps_get_minimum_free_size" is not so free ... -The actual firmware are not stable : I have sometimes unwanted reboots and my devices get slower and slower until a reboot. I mean there is many memory leaks -Himem no malloc() need to alloc a range of memory, setting it that way cannot lead to memory leakage. It's a bit like the partition table. You do not write in a part that is used by other. But it's more complicated and need a global definition, perhaps in defines.h +No emergency to implement HIMEM, need to be discussed and compared with existing implementation.

I've pulled HIMEM tools in PR https://github.com/jomjol/AI-on-the-edge-device/pull/1852 for testing purpose.

Many thanks for your professionnal comments

Regards

[Slider0007]

-Your heap output "Int Min Free: 13879" aka "heap_caps_get_minimum_free_size" is not so free ...

How do you know this? Do you know a better routine to show lowest available memory?

-The actual firmware are not stable : I have sometimes unwanted reboots and my devices get slower and slower until a reboot. I mean there is many memory leaks

I agree partially, with latest rolling I do not have any slowdown / reboot issues anymore. For sure, a longer testrun has to prove this. There could still be memory leackages in there. Quality can be improved all the time, but it needs time which is unfortunately limited (at least for me).

[caco3]

Thank you both for your investigations!

If there is a memory leak, we should investigate. Since there are so many config options available, that can be a tricky part. My 2 devices where running for 9 days with interval of 2 minutes. They only restarted because I told them or because of https://github.com/jomjol/AI-on-the-edge-device/pull/1837 which I hope we can get rid off.

A side topic: @slider as well as @nliaudat I think we should document all that learnings on a documentation page, also the REST API and compile flags.

[Slider0007]

@slider as well as @nliaudat I think we should document all that learnings on a documentation page, also the REST API and compile flags.

--> REST API heap handler added to documentation

[nliaudat]

I prepare a memory leak detection using esp-idf heap debug

Output is like

16 bytes (@ 0x3ffd1914) allocated CPU 1 ccount 0x419d1cf8 caller 0x4010a09b:0x4010a0c6
16 bytes (@ 0x3ffd18cc) allocated CPU 1 ccount 0x419d5680 caller 0x4010a09b:0x4010a0c6
78141 bytes 'leaked' in trace (96 allocations)
total allocations 5598 total frees 5447

Adresses can be decoded with idf monitor

Done : https://github.com/jomjol/AI-on-the-edge-device/pull/1861