espressif / esp-idf

Espressif IoT Development Framework. Official development framework for Espressif SoCs.
Apache License 2.0
13.65k stars 7.29k forks source link

Statically allocated DRAM is limited to 160KB (IDFGH-1187) #3497

Closed dsyleixa closed 1 year ago

dsyleixa commented 5 years ago

code compiles and works for M4 plus TFT HX8357, but fails for ESP32; Arduino IDE 1.8.8 Board: Adafruit Feather ESP32
core: ESP32 espressive 1.0.2

compile errors (when linking) for my code

Arduino: 1.8.8 (Windows 7), Board: "Adafruit ESP32 Feather, 80MHz, 921600, None"

D:\arduino\arduino-builder -dump-prefs -logger=machine -hardware D:\arduino\hardware -hardware D:\arduino\portable\packages -tools D:\arduino\tools-builder -tools D:\arduino\hardware\tools\avr -tools D:\arduino\portable\packages -built-in-libraries D:\arduino\libraries -libraries D:\arduino\portable\sketchbook\libraries -fqbn=esp32:esp32:featheresp32:FlashFreq=80,UploadSpeed=921600,DebugLevel=none -ide-version=10808 -build-path C:\Users\hw\AppData\Local\Temp\arduino_build_226627 -warnings=none -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.esptool_py.path=D:\arduino\portable\packages\esp32\tools\esptool_py\2.6.1 -prefs=runtime.tools.esptool_py-2.6.1.path=D:\arduino\portable\packages\esp32\tools\esptool_py\2.6.1 -prefs=runtime.tools.mkspiffs.path=D:\arduino\portable\packages\esp32\tools\mkspiffs\0.2.3 -prefs=runtime.tools.mkspiffs-0.2.3.path=D:\arduino\portable\packages\esp32\tools\mkspiffs\0.2.3 -prefs=runtime.tools.xtensa-esp32-elf-gcc.path=D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0 -prefs=runtime.tools.xtensa-esp32-elf-gcc-1.22.0-80-g6c4433a-5.2.0.path=D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0 -verbose D:\Akten\Programmierung\ArduinoProgs\ConwaysGameOfLife\ConwayGoL_M4_Logic_NOR\ConwayGoL_M4_Logic_NOR.ino D:\arduino\arduino-builder -compile -logger=machine -hardware D:\arduino\hardware -hardware D:\arduino\portable\packages -tools D:\arduino\tools-builder -tools D:\arduino\hardware\tools\avr -tools D:\arduino\portable\packages -built-in-libraries D:\arduino\libraries -libraries D:\arduino\portable\sketchbook\libraries -fqbn=esp32:esp32:featheresp32:FlashFreq=80,UploadSpeed=921600,DebugLevel=none -ide-version=10808 -build-path C:\Users\hw\AppData\Local\Temp\arduino_build_226627 -warnings=none -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.esptool_py.path=D:\arduino\portable\packages\esp32\tools\esptool_py\2.6.1 -prefs=runtime.tools.esptool_py-2.6.1.path=D:\arduino\portable\packages\esp32\tools\esptool_py\2.6.1 -prefs=runtime.tools.mkspiffs.path=D:\arduino\portable\packages\esp32\tools\mkspiffs\0.2.3 -prefs=runtime.tools.mkspiffs-0.2.3.path=D:\arduino\portable\packages\esp32\tools\mkspiffs\0.2.3 -prefs=runtime.tools.xtensa-esp32-elf-gcc.path=D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0 -prefs=runtime.tools.xtensa-esp32-elf-gcc-1.22.0-80-g6c4433a-5.2.0.path=D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0 -verbose D:\Akten\Programmierung\ArduinoProgs\ConwaysGameOfLife\ConwayGoL_M4_Logic_NOR\ConwayGoL_M4_Logic_NOR.ino Using board 'featheresp32' from platform in folder: D:\arduino\portable\packages\esp32\hardware\esp32\1.0.2 Using core 'esp32' from platform in folder: D:\arduino\portable\packages\esp32\hardware\esp32\1.0.2

...

Linking everything together... "D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0/bin/xtensa-esp32-elf-gcc" -nostdlib "-LD:\arduino\portable\packages\esp32\hardware\esp32\1.0.2/tools/sdk/lib" "-LD:\arduino\portable\packages\esp32\hardware\esp32\1.0.2/tools/sdk/ld" -T esp32_out.ld -T esp32.common.ld -T esp32.rom.ld -T esp32.peripherals.ld -T esp32.rom.spiram_incompatible_fns.ld -u ld_include_panic_highint_hdl -u call_user_start_cpu0 -Wl,--gc-sections -Wl,-static -Wl,--undefined=uxTopUsedPriority -u cxa_guard_dummy -u cxx_fatal_exception -Wl,--start-group "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\sketch\ConwayGoL_M4_Logic_NOR.ino.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Wire\Wire.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\SPI\SPI.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit-GFX-Library-master\Adafruit_GFX.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit-GFX-Library-master\Adafruit_SPITFT.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit-GFX-Library-master\glcdfont.c.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit_HX8357_Library-master\Adafruit_HX8357.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit_STMPE610-master\Adafruit_STMPE610.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\core\core.a" -lgcc -lopenssl -lbtdm_app -lfatfs -lwps -lcoexist -lwear_levelling -lesp_http_client -lprotobuf-c -lhal -lnewlib -ldriver -lbootloader_support -lpp -lfreemodbus -lmesh -lsmartconfig -ljsmn -lwpa -lethernet -lphy -lfrmn -lapp_trace -lfr_coefficients -lconsole -lulp -lwpa_supplicant -lfreertos -lbt -lmicro-ecc -lesp32-camera -lcxx -lxtensa-debug-module -ltcp_transport -lmdns -lvfs -lmtmn -lesp_ringbuf -lsoc -lcore -lfb_gfx -lsdmmc -llibsodium -lcoap -ltcpip_adapter -lprotocomm -lesp_event -limage_util -lc_nano -lesp-tls -lasio -lrtc -lspi_flash -lwpa2 -lwifi_provisioning -lesp32 -lface_recognition -lapp_update -lnghttp -lspiffs -lface_detection -lespnow -lnvs_flash -lesp_adc_cal -llog -ldl_lib -lsmartconfig_ack -lexpat -lfd_coefficients -lm -lmqtt -lc -lheap -lmbedtls -llwip -lnet80211 -lesp_http_server -lpthread -ljson -lesp_https_ota -lstdc++ -Wl,--end-group -Wl,-EL -o "C:\Users\hw\AppData\Local\Temp\arduino_build_226627/ConwayGoL_M4_Logic_NOR.ino.elf" d:/arduino/portable/packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: C:\Users\hw\AppData\Local\Temp\arduino_build_226627/ConwayGoL_M4_Logic_NOR.ino.elf section .dram0.bss' will not fit in regiondram0_0_seg'

d:/arduino/portable/packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: DRAM segment data does not fit.

d:/arduino/portable/packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: region `dram0_0_seg' overflowed by 74776 bytes

collect2.exe: error: ld returned 1 exit status

Bibliothek Wire in Version 1.0.1 im Ordner: D:\arduino\portable\packages\esp32\hardware\esp32\1.0.2\libraries\Wire wird verwendet Bibliothek SPI in Version 1.0 im Ordner: D:\arduino\portable\packages\esp32\hardware\esp32\1.0.2\libraries\SPI wird verwendet Bibliothek Adafruit-GFX-Library-master in Version 1.4.10 im Ordner: D:\arduino\portable\sketchbook\libraries\Adafruit-GFX-Library-master wird verwendet Bibliothek Adafruit_HX8357_Library-master in Version 1.1.4 im Ordner: D:\arduino\portable\sketchbook\libraries\Adafruit_HX8357_Library-master wird verwendet Bibliothek Adafruit_STMPE610-master in Version 1.0.1 im Ordner: D:\arduino\portable\sketchbook\libraries\Adafruit_STMPE610-master wird verwendet exit status 1 Fehler beim Kompilieren für das Board Adafruit ESP32 Feather.

complete error msg in attachement compileError.txt

source code also attached ConwayGoL_M4_Logic_NOR.zip

negativekelvin commented 5 years ago

Yes too much static ram usage. This is the wrong place to post

Read this https://github.com/espressif/arduino-esp32/issues/1163 Ask for help here https://esp32.com/viewforum.php?f=19

dsyleixa commented 5 years ago

hmmm - that's weird:

ESP32: 4 MByte flash, 520 KB SRAM https://learn.adafruit.com/adafruit-huzzah32-esp32-feather/overview

M4: 512 KB flash, 192 KB RAM https://learn.adafruit.com/adafruit-huzzah32-esp32-feather/overview

if ESP32 RAM is so restricted, then that is big drawback and that issue should be fixed ASAP!

dsptech commented 5 years ago

from https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/system/mem_alloc.html

"Note: Due to a technical limitation, the maximum statically allocated DRAM usage is 160KB. The remaining 160KB (for a total of 320KB of DRAM) can only be allocated at runtime as heap."

dsyleixa commented 5 years ago

OMG, that should have been made absolutely clear BEFORE I had purchased that thing! OTOH, Adafruit advertises the ESP by 520 KB SRAM

Can that "technical limitation" be reworked and fixed?

projectgus commented 5 years ago

Hi @dsyleixa,

The total SRAM is 520KB but this is DRAM+IRAM, and only DRAM (320KB) is normally used for data, IRAM is normally used for executable instructions. It is possible to store data in IRAM but there are restrictions.

Regarding how Adafruit advertise the ESP32, you'll have to take this up with them. But they are technically correct.

Working around the static memory limitation should be possible without too much work. If you have some larger buffer like this:

static uint8_t framebuffer[131072];

Then change it to a pointer, and allocate it during setup with malloc() or calloc(). This makes it dynamic memory not static memory:

static uint8_t *framebuffer;

void setup() {
   framebuffer = calloc(1, 131072);
   assert(framebuffer != NULL);

   // do other setup stuff
}

Also, as @negativekelvin mentions, for Arduino issues please post on the Arduino issue tracker or the Arduino forum of esp32.com.

dsyleixa commented 5 years ago

sorry, but I can't handle pointer and allocates, that is far beyond my skills :(

I don't post to Arduino.cc forum, because there are too many outrageous and insulting users.

projectgus commented 5 years ago

There is an Arduino subforum on esp32.com, that's the one I was referring to: https://esp32.com/viewforum.php?f=19

dsyleixa commented 5 years ago

ok, perhaps in future. As to now, to me it's an ESP32 hardware issue which should be reworked and fixed, not discussed ;)

projectgus commented 5 years ago

There's no hardware issue. This is a software issue in ESP-IDF that the amount of statically assigned DRAM in an app is limited to 160KB. We don't have a GitHub issue tracking this, so we can start tracking it here.

We don't have an ETA for fixing this issue as it's complex to fix (for architectural reasons) and it's generally easy to work around by assigning large buffers dynamically (see above post with suggestion). But we do plan to fix it at some point.

(There is also the related but different issue #3211 )

galah92 commented 4 years ago

I'm also looking for a solution for that, dynamic allocation makes my code much more error-prone.

SoucheSouche commented 1 year ago

The DRAM limitations are now clearly stated in the documentation.

zikalino commented 1 year ago

closing the issue

dsyleixa commented 1 year ago

my point is to change 2x160kB to 1x320kB to be able to address it as a unit in its entirety. My point is not about documentation or discussion.

dsyleixa commented 1 year ago

please reopen this issue because it is not fixed yet.

SoucheSouche commented 1 year ago

Hi @dsyleixa,

I am sorry we closed the issue after more than 2 years of inactivity without giving out more information about why we closed it.

The reason why ~2x160KB of DRAM is available instead of the ~1x320KBon esp32 is because some functions used during startup are stored starting from the address 0x3ffexxxx which happens to be in the middle of the DRAM section starting at 0x3ffbxxxx making it not continuous and preventing you from creating static arrays bigger than approx. 160KB.

Fixing the problem would require moving said functions at the end of the DRAM memory but as @projectgus mentioned more than 2 years ago this is far from being a trivial fix and it would require a lot of architectural changes which explains why as of now it has not been addressed.

As I can't guaranty that this issue will ever be fixed, I would encourage you to consider (or reconsider) the following workaround:

We have an internal ticket reporting this issue so it won't be forgotten and we don't want you to rely on this issue being fixed to continue with your project development. For those reason I would recommend against reopening it but feel free to let me know if you disagree with this decision now that I provided more information about it.

I am sorry again that this issue was left unaddressed for so long.