Closed KonssnoK closed 6 months ago
This is just a tip to narrow down the source of the difference: You can use esp-idf-size (https://github.com/espressif/esp-idf-size) (already available in ESP-IDF environments) and with the --diff
argument to see the difference between the two projects.
python -m esp_idf_size.ng build/blink_v5.1.map --diff /tmp/blink_v4.4.map
You can add arguments to get more details. See python -m esp_idf_size.ng --help
.
This is just a tip to narrow down the source of the difference: You can use esp-idf-size (https://github.com/espressif/esp-idf-size) (already available in ESP-IDF environments) and with the
--diff
argument to see the difference between the two projects.python -m esp_idf_size.ng build/blink_v5.1.map --diff /tmp/blink_v4.4.map
You can add arguments to get more details. See
python -m esp_idf_size.ng --help
.
hello @dobairoland , we are talking about internal RAM usage, not file size. I'll update the name to reflect that
After quite some debugging I think I have found the difference between release/v4.4
and release/v5.1
that is causing the above mentioned behavior.
While in v4.4
the FreeRTOS malloc was simply defined as #define pvPortMalloc malloc
, in v5.1
it is now defined as
#define portFREERTOS_HEAP_CAPS ( MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT )
/*-----------------------------------------------------------*/
void * pvPortMalloc( size_t xWantedSize )
{
void * pvReturn = NULL;
/* All dynamic allocation done by FreeRTOS goes through this function. If
* users need to allocate FreeRTOS objects into external RAM, they should
* use the "static" equivalents of FreeRTOS API to create FreeRTOS objects
* (e.g., queues). */
pvReturn = heap_caps_malloc( xWantedSize, portFREERTOS_HEAP_CAPS );
return pvReturn;
}
which forces the allocations to happen on the internal RAM.
By placing logs in esp_wifi/esp32s3/esp_adapter.c
I can observe that calling esp_mesh_start()
creates a queue of 96 elements with 184 bytes each (resulting in an allocation of 17664 bytes). By changing the code in queue_create_wrapper
I managed to allocate the storage of the queue in SPIRAM:
- return (void *)xQueueCreate(queue_len, item_size);
+ return (void *)xQueueCreateWithCaps(queue_len, item_size, MALLOC_CAP_SPIRAM);
I tried to apply a similar modification to task_create_wrapper
, however it fails on an assertion when creating a task with the name wifi
:
assert failed: spi_flash_disable_interrupts_caches_and_other_cpu cache_utils.c:154 (esp_task_stack_is_sane_cache_disabled())
I assume this is related to the explanation in the code, but I am not sure I 100% understand the comment
/*
* All dynamic allocation done by FreeRTOS should be placed in internal 8-bit
* accessible RAM (i.e., using the MALLOC_CAP_INTERNAL|MALLOC_CAP_8BIT flags).
* This is due to the fact that FreeRTOS objects (e.g., task stacks, TCBs,
* queues etc) must be accessible even if the cache is disabled. Therefore, the
* heap that is made available to FreeRTOS for dynamic allocation is a subset of
* the ESP-IDF heap (where all MALLOC_CAP_INTERNAL|MALLOC_CAP_8BIT memory is
* made available to FreeRTOS for dynamic allocation).
* */
what is not clear is why we can write to spi_flash in v4.4 while having tasks allocated in the spi_ram and we cannot do the same in 5.1.
For sure there are no hardware limitations, or we would not have been abled in the first place to put our RTOS stuff on SPI_RAM.
@dobairoland @Dazza0 PTAL
@KonssnoK Please take a look at the FreeRTOS section of the IDF v5.1 migration guide. All dynamic memory allocated by FreeRTOS will not default to be being internal memory. This change was intentional in to increase run time safety.
For library/user code that need to move memory allocations to external memory. This now needs to be done explicitly (see the CreateWithCaps()
API mentioned in the migration guide).
@KonssnoK Please take a look at the FreeRTOS section of the IDF v5.1 migration guide. All dynamic memory allocated by FreeRTOS will not default to be being internal memory. This change was intentional in to increase run time safety.
For library/user code that need to move memory allocations to external memory. This now needs to be done explicitly (see the
CreateWithCaps()
API mentioned in the migration guide).
Hello @Dazza0 , could you please be more specific on what you mean with "increase runtime safety" ?
Does this mean that 4.4 is not runtime safe and the change needs to be backported?
What is the criticality? Is it related to the HW design?
Thanks.
@KonssnoK Previously, FreeRTOS would simply use malloc
. Thus, if users enabled CONFIG_SPIRAM_USE_MALLOC
, it is possible that malloc()
will return memory in SPIRAM. Thus, it was possible that xQueueCreate()
could allocate a queue in SPIRAM. Given that SPIRAM is inaccessible when the cache is disabled (e.g., during an SPI flash write), it was up to the users to ensure that FreeRTOS objects that were allocated in SPIRAM were not access while the cache was disabled. For example, if
xQueueCreate()
allocates a queue in SPIRAM, users must ensure that the queue was not accessed while another task was executing an SPI flash write operation.
However, we encountered numerous users who ran into a Cache disabled but cached memory region accessed
error as they were not aware of the restrictions of allocating memory from SPIRAM. Thus, the decision was made to make FreeRTOS to allocate all dynamic memory from internal memory by default (which is always accessible). FreeRTOS objects can still be allocated from SPIRAM (or other types of memory as well). However, this must now be done explicitly by the user (via the ...WithCaps()
API) such that they are aware of exactly which type of memory the object is placed in, and the associated restrictions.
thanks @Dazza0 .
Am I correct saying that the only way to have such a scenario of Cache disabled but cached memory region accessed
would be when the second core tries to do something on spi ram during spi flash operations?
Because from what I remember on flash operations you always have at least
@KonssnoK
Am I correct saying that the only way to have such a scenario of Cache disabled but cached memory region accessed would be when the second core tries to do something on spi ram during spi flash operations?
Yes, that's correct
Because from what I remember on flash operations you always have at least
scheduler is suspended on running CPU interrupts are disabled
Interrupts created with the ESP_INTR_FLAG_IRAM
will still run during flash operations. If that interrupt handler then calls xQueueSendFromISR()
to an SPI RAM queue, we get the Cache disabled but cached memory region accessed
error.
our main doubt is about some espressif closed source code (mesh) that we use extensively.
This means that to recover space we'll have to change the esp_adapter file to use withCaps
but we have no visibility on what happens inside the mesh library.
@zhangyanjiaoesp could you make an evaluation of:
ESP_INTR_FLAG_IRAM
interrupts there are in the mesh library.xQueueSendFromISR
in the mesh libraryIf 1 is true (used) it means that we should not move mesh queues to spi ram (even if we currently do).
If 1 is false (not used) it means we can use the mesh library on the same core of spi flash operations independently from 2, which somehow invalidates https://github.com/espressif/esp-idf/issues/11023
If 1 is false and 2 false (not used) it means we can also move mesh to the second core.
@Dazza0 please confirm.
Thanks
@zhangyanjiaoesp could you make an evaluation of:
- how many
ESP_INTR_FLAG_IRAM
interrupts there are in the mesh library.- if there are
xQueueSendFromISR
in the mesh library
@KonssnoK
ESP_INTR_FLAG_IRAM
interrupts in the mesh libraryxQueueSendFromISR
in the mesh library@zhangyanjiaoesp thanks!
so it means we should be able to change esp_adapter, add withCaps
and move it to core1 without any issue.
Hi, I'm facing the same issue: my project in v5.1 is consuming internal RAM more than same code in v4.4 and always the compilation return "out of region..." Which are the steps to solve this issue?
@romocitto88 Please take a look at https://github.com/espressif/esp-idf/issues/13285#issuecomment-1975891933 and https://github.com/espressif/esp-idf/issues/13285#issuecomment-1978103060
I read the comments and I changed the functions indicated with "...WithCaps() with MALLOC_CAP_SPIRAM" but nothing is changed. I missed something?
@romocitto88 Could you post a copy of the error log you are encountering?
@Dazza0 Here the error:
c:/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/12.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: .pio/build/esp32dev/firmware.elf section
.iram0.text' will not fit in region
iram0_0_seg' c:/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/12.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: IRAM0 segment data does not fit. c:/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/12.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: region `iram0_0_seg' overflowed by 2528 bytes
@romocitto88 You are running our of static memory (IRAM to be specific). It's completely unrelated to this issue.
Please refer to this guide regarding reducing binary size.
@Dazza0 The same project with ESP-IDF version 4.4.2 everything compiles without problems, and with ESP-IDF version 5.1.2 runs out the memory. I've attached a file merge of the two section compiled with the different IDF-VERSION Merge.pdf
@romocitto88 Yes, more functions have been placed into IRAM between v4.4.2 and v5.1.2. Again, please refer to the reducing binary size guide to determine what is using up your IRAM and what can be moved to flash to reduce IRAM usage.
Answers checklist.
General issue report
this was tested on esp32s3 devkit-C using https://github.com/espressif/esp-idf/tree/master/examples/mesh/ip_internal_network
Changes on v4.4
sdkconfig
Log on v4.4
Changes in 5.1
esp-idf release/v5.1
sdkconfig
Log
See last line of log: v4.4: 245207 bytes free v5.1: 214471 bytes free -> delta ~ 30.7k
FYI @zhangyanjiaoesp