PSRAM enabled, but not usable as internal heap memory

cyberman54 commented 5 years ago

How can use PSRAM as internal heap memory?

I saw this already working in previous releases, PSRAM was automatically added to internal heap memory when device started. But since latest updates this is broken or changed in some way.

[D][esp32-hal-psram.c:47] psramInit(): PSRAM enabled
[I][main.cpp:109] setup(): Internal Total heap 239056, internal Free Heap 213212
[I][main.cpp:111] setup(): SPIRam Total heap 4194252, SPIRam Free Heap 4194252

lbernstone commented 5 years ago

The behavior of ESP.getHeapSize was incorrect when PSRAM support was first added (~Feb 2018). PSRAM is not directly part of your heap. So, the functions were reverted in August to just show the free/total SRAM, and PSRAM got its own functions. If you want to allocate memory from PSRAM, you will need to use ps_malloc manually (as was always the case).

cyberman54 commented 5 years ago

@ibernstone I need to store a std::set container in PSRAM. Not sure how to do this. I thought the PSRAM would increase the heap where std::set normally stores it's contents, and i was convinced that this worked automatically before on boards which has PSRAM? I could see ESP.getHeapSize couting down from a high number including PSRAM size, while storing elements in a std::set container.

lbernstone commented 5 years ago

Dynamic objects like a set will need a custom allocator. I don't have an example for you, though I do think it would be quite a useful thing. Maybe one of the big C brains out there can make an example.

cyberman54 commented 5 years ago

Allright, thank you for your help. Since it turned out that this is not an issue of this library, i close this issue here.

me-no-dev commented 5 years ago

Arduino is compiled with the option to not add PSRAM to the general heap, because most Arduino users have no idea what is the difference between internal RAM and PSRAM and will allocate everything there, then use it in interrupts and have all kinds of issues and errors. Therefore we have provided the ps_malloc and friends to allocate memory there when needed.

sonolive commented 2 years ago

Arduino is compiled with the option to not add PSRAM to the general heap,

hello, ok, but how to allow this option for a new build of arduino ??? thx in advance olive

lbernstone commented 2 years ago

Since v2.0.0, psram has been directly mapped, so allocations > 16k (and 4k in the newer chips) are created in psram. While I am here, I might as well post code for a custom allocator, which I don't think I have posted before.

template <class T>
struct PSallocator {
  typedef T value_type;
  PSallocator() = default;
  template <class U> constexpr PSallocator(const PSallocator<U>&) noexcept {}
  [[nodiscard]] T* allocate(std::size_t n) {
    if(n > std::size_t(-1) / sizeof(T)) throw std::bad_alloc();
    if(auto p = static_cast<T*>(ps_malloc(n*sizeof(T)))) return p;
    throw std::bad_alloc();
  }
  void deallocate(T* p, std::size_t) noexcept { std::free(p); }
};
template <class T, class U>
bool operator==(const PSallocator<T>&, const PSallocator<U>&) { return true; }
template <class T, class U>
bool operator!=(const PSallocator<T>&, const PSallocator<U>&) { return false; }

std::vector<int, PSallocator<int> > v;

void setup() {
  Serial.begin(115200);
  delay(1000);
  Serial.println(ESP.getFreePsram());
  v.reserve(900000);
  for (uint32_t x=0; x<900000; x++) {
    v.push_back(x);
  }
  Serial.println(ESP.getFreePsram());
  v.clear();
  v.shrink_to_fit();
  Serial.println(ESP.getFreePsram());
}

sonolive commented 2 years ago

hello lbernstone

many thanks for the answer, but what i would like to achieve is something like this :


const int size_tablo = 150000;
EXT_RAM_ATTR int test_tablo_int[size_tablo];

void setup() {
Serial.begin(115200);

  Serial.print("Build test_tablo_int - size = ");Serial.println(size_tablo);

  for(int i = 0 ; i < size_tablo ; i++)
  {
  test_tablo_int[i] = i;
  }
  Serial.print("size of test_tablo_int : ");Serial.print(float(sizeof(test_tablo_int))/1000,3);Serial.println(" KB");

 Serial.println(ESP.getFreePsram());
}

i tried to compile with

#
# SPI RAM config
#
CONFIG_SPIRAM=y
CONFIG_SPIRAM_BOOT_INIT=y
CONFIG_SPIRAM_USE_MALLOC=y
CONFIG_SPIRAM_MEMTEST=y
CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY=y
CONFIG_SPIRAM_CACHE_WORKAROUND=y

in eclipse espressif IDE ( arduino as esp-idf component ) it works well

i compared the two builds : CONFIG_SPIRAM_BOOT_INIT is not set CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY=y versus CONFIG_SPIRAM_BOOT_INIT=y CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY=y with WinMerge and only libesp_system.a and libapp_update.a were different ... and also libarduino.a ...

then copied/pasted the libesp_system.a and libapp_update.a from eclipse espressif IDE : C:\Espressif\frameworks\esp-idf-v4.4\workspace\blink_2\build\esp-idf\esp_system and C:\Espressif\frameworks\esp-idf-v4.4\workspace\blink_2\build\esp-idf\app_update to : ...\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.2\tools\sdk\esp32\lib

then compiled with Arduino IDE

it reboots and the error is : [E][esp32-hal-psram.c:76] psramInit(): PSRAM could not be added to the heap!

in many cases i have tested for few days, the line CONFIG_SPIRAM_BOOT_INIT=y causes the issues in arduino IDE

lbernstone commented 2 years ago

psram is not static ram- it is not available until the system is initialized. You cannot create global variables there. Create a static object when you first need to use the psram variable.

sonolive commented 2 years ago

hello lbernstone thank you for the answer, Maybe my explainations were not clear enough ...

i will try to do better ;-)

You cannot create global variables there

I CAN : this works very well with eclipse ESP-IDE

with this SPI RAM CONFIG in sdkconfig ->

#
# SPI RAM config
#
CONFIG_SPIRAM_TYPE_AUTO=y
# CONFIG_SPIRAM_TYPE_ESPPSRAM16 is not set
# CONFIG_SPIRAM_TYPE_ESPPSRAM32 is not set
# CONFIG_SPIRAM_TYPE_ESPPSRAM64 is not set
CONFIG_SPIRAM_SIZE=-1
CONFIG_SPIRAM_SPEED_40M=y
CONFIG_SPIRAM=y
CONFIG_SPIRAM_BOOT_INIT=y
# CONFIG_SPIRAM_USE_MEMMAP is not set
# CONFIG_SPIRAM_USE_CAPS_ALLOC is not set
CONFIG_SPIRAM_USE_MALLOC=y
CONFIG_SPIRAM_MEMTEST=y
CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=4096
CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y
CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=0
CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY=y
# CONFIG_SPIRAM_ALLOW_NOINIT_SEG_EXTERNAL_MEMORY is not set
CONFIG_SPIRAM_CACHE_WORKAROUND=y

this code ->

#include "Arduino.h"

void memories (String str);

int free_CAP_8BIT;
int free_CAP_SPIRAM;
int free_CAP_INTERNAL;

const int size_tablo = 150000;
EXT_RAM_ATTR int test_tablo_int[size_tablo];

void setup(){
    Serial.begin(115200);
        for(int i = 0 ; i < size_tablo ; i++){
            test_tablo_int[i] = i;
            }
    Serial.print("size of test_tablo_int : ");Serial.print(float(sizeof(test_tablo_int))/1000,3);Serial.println(" KB");
    test_tablo_int[size_tablo / 10] = 1;
    test_tablo_int[size_tablo / 5] = 2;
    test_tablo_int[size_tablo / 2] = 3;

        for(int i = 0 ; i < size_tablo ; i+=(size_tablo/10)){
             Serial.print("tablo[");Serial.print(i);Serial.print("] = ");Serial.println(test_tablo_int[i]);
            }
    memories ("memories @ end setup");
    Serial.println("end setup");
}

void loop(){
//  Serial.println("loop");
/*
    for(int i = 0 ; i < size_tablo ; i+=(size_tablo/10))
      {
          Serial.print("tablo[");Serial.print(i);Serial.print("] = ");Serial.println(test_tablo_int[i]);
          delay(3000);
      }
*/
}

void memories (String str)
{
  delay(10);
  Serial.print("\n------------------\n");Serial.println(str);
  multi_heap_info_t info;
  heap_caps_get_info(&info, MALLOC_CAP_8BIT);
  free_CAP_8BIT = info.total_free_bytes ;
  heap_caps_get_info(&info, MALLOC_CAP_SPIRAM);
  free_CAP_SPIRAM = info.total_free_bytes ;
  free_CAP_INTERNAL = info.total_free_bytes ;
  Serial.print(">>>> FREE\n");
  Serial.print("MALLOC_CAP_8BIT : ");Serial.println(float(free_CAP_8BIT)/1000,3);
  Serial.print("MALLOC_CAP_SPIRAM : ");Serial.println(float(free_CAP_SPIRAM)/1000,3);
  Serial.print("MALLOC_CAP_INTERNAL : ");Serial.println(float(free_CAP_INTERNAL)/1000,3);
  Serial.print("---------------------\n");
}

outputs ->

------------------
size of test_tablo_int : 600.000 KB
tablo[0] = 0
tablo[15000] = 1
tablo[30000] = 2
tablo[45000] = 45000
tablo[60000] = 60000
tablo[75000] = 3
tablo[90000] = 90000
tablo[105000] = 105000
tablo[120000] = 120000
tablo[135000] = 135000
------------------
memories @ end setup
>>>> FREE
MALLOC_CAP_8BIT : 3861.203
MALLOC_CAP_SPIRAM : 3592.139
MALLOC_CAP_INTERNAL : 338.216
---------------------
end setup

so ... everything is OK ! free SPIRAM is : 4192.139 - 600.000 = 3592.139 as excpected

const int size_tablo = 150000;
EXT_RAM_ATTR int test_tablo_int[size_tablo];

these lines DO ALLOCATE 600.000 KB in EXT_RAM

the same code with different size of "const int size_tablo" also give good results ;

this code ->

const int size_tablo = 1000;
EXT_RAM_ATTR int test_tablo_int[size_tablo];

void setup(){
    Serial.begin(115200);
        for(int i = 0 ; i < size_tablo ; i++){
            test_tablo_int[i] = i;
            }
    Serial.print("size of test_tablo_int : ");Serial.print(float(sizeof(test_tablo_int))/1000,3);Serial.println(" KB");
    test_tablo_int[size_tablo / 10] = 1;
    test_tablo_int[size_tablo / 5] = 2;
    test_tablo_int[size_tablo / 2] = 3;

        for(int i = 0 ; i < size_tablo ; i+=(size_tablo/10)){
             Serial.print("tablo[");Serial.print(i);Serial.print("] = ");Serial.println(test_tablo_int[i]);
            }
    memories ("memories @ end setup");
    Serial.println("end setup");
}

outputs ->

------------------
size of test_tablo_int : 4.000 KB
tablo[0] = 0
tablo[100] = 1
tablo[200] = 2
tablo[300] = 300
tablo[400] = 400
tablo[500] = 3
tablo[600] = 600
tablo[700] = 700
tablo[800] = 800
tablo[900] = 900
------------------
memories @ end setup
>>>> FREE
MALLOC_CAP_8BIT : 4457.203
MALLOC_CAP_SPIRAM : 4188.139
MALLOC_CAP_INTERNAL : 338.216
---------------------
end setup

and it is STILL OK ... free SPIRAM is : 4192.139 - 4.000 = 4188.139 as excpected we can change the value of "const int size_tablo" as you want (w/ the max limit of 4MB of course ...)

with this SPI RAM CONFIG i can also alloc via ps_malloc or the custom allocator you shared above

The issue is when i try to compile this code, with the same SPI RAM CONFIG in ARDUINO IDE (I copied/pasted the C:\Espressif\frameworks\esp-idf-v4.4\workspace\blink_2\build\esp-idf\esp_system\libesp_system.a into ...\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.2\tools\sdk\esp32\lib )

with ->

const int size_tablo = 1000;
EXT_RAM_ATTR int test_tablo_int[size_tablo];

the output is ->

11:12:22.935 -> ets Jun  8 2016 00:22:57
11:12:22.935 -> 
11:12:22.935 -> rst:0x1 (POWERON_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
11:12:22.935 -> configsip: 0, SPIWP:0xee
11:12:22.935 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
11:12:22.935 -> mode:DIO, clock div:1
11:12:22.935 -> load:0x3fff0030,len:1184
11:12:22.935 -> load:0x40078000,len:12804
11:12:22.935 -> ho 0 tail 12 room 4
11:12:22.982 -> load:0x40080400,len:3032
11:12:22.982 -> entry 0x400805e4
11:12:24.808 -> [   862][E][esp32-hal-psram.c:76] psramInit(): PSRAM could not be added to the heap!
11:12:24.856 -> size of test_tablo_int : 4.000 KB
11:12:24.856 -> tablo[0] = 0
11:12:24.856 -> tablo[100] = 1
11:12:24.902 -> tablo[200] = 2
11:12:24.902 -> tablo[300] = 300
11:12:24.902 -> tablo[400] = 400
11:12:24.902 -> tablo[500] = 3
11:12:24.902 -> tablo[600] = 600
11:12:24.902 -> tablo[700] = 700
11:12:24.902 -> tablo[800] = 800
11:12:24.902 -> tablo[900] = 900
11:12:24.902 -> >>>>>> FREE
11:12:24.902 -> Total PSRAM : 0.000
11:12:24.902 -> free PSRAM : 0.000
11:12:24.902 -> 
11:12:24.902 -> end setup

and i cannot use the function void memories (String str); -> it reboots

1:18:31.676 -> ets Jun  8 2016 00:22:57
11:18:31.676 -> 
11:18:31.676 -> rst:0xc (SW_CPU_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
11:18:31.676 -> configsip: 0, SPIWP:0xee
11:18:31.676 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
11:18:31.676 -> mode:DIO, clock div:1
11:18:31.676 -> load:0x3fff0030,len:1184
11:18:31.676 -> load:0x40078000,len:12804
11:18:31.676 -> ho 0 tail 12 room 4
11:18:31.676 -> load:0x40080400,len:3032
11:18:31.724 -> entry 0x400805e4
11:18:33.556 -> [   862][E][esp32-hal-psram.c:76] psramInit(): PSRAM could not be added to the heap!
11:18:33.604 -> size of test_tablo_int : 4.000 KB
11:18:33.604 -> tablo[0] = 0
11:18:33.604 -> tablo[100] = 1
11:18:33.604 -> tablo[200] = 2
11:18:33.604 -> tablo[300] = 300
11:18:33.604 -> tablo[400] = 400
11:18:33.604 -> tablo[500] = 3
11:18:33.604 -> tablo[600] = 600
11:18:33.604 -> tablo[700] = 700
11:18:33.604 -> tablo[800] = 800
11:18:33.651 -> tablo[900] = 900
11:18:33.651 -> 
11:18:33.651 -> ------------------
11:18:33.651 -> memories @ end setup
11:18:33.651 -> Guru Meditation Error: Core  1 panic'ed (StoreProhibited). Exception was unhandled.
11:18:33.651 -> 
11:18:33.651 -> Core  1 register dump:
11:18:33.651 -> PC      : 0x4008b330  PS      : 0x00060f33  A0      : 0x8008d030  A1      : 0x3ffb26b0  
11:18:33.651 -> A2      : 0xaaaaaaaa  A3      : 0xb33fffff  A4      : 0x0000cdcd  A5      : 0x00060f23  
11:18:33.651 -> A6      : 0x00060f20  A7      : 0x0000abab  A8      : 0x0000abab  A9      : 0xffffffff  
11:18:33.651 -> A10     : 0x00000003  A11     : 0x00060f23  A12     : 0x00060f20  A13     : 0x00000000  
11:18:33.698 -> A14     : 0x6b2aaaaa  A15     : 0x003fffff  SAR     : 0x00000014  EXCCAUSE: 0x0000001d  
11:18:33.698 -> EXCVADDR: 0xaaaaaaaa  LBEG    : 0x40086b4e  LEND    : 0x40086b59  LCOUNT  : 0x00000000  
11:18:33.698 -> 
11:18:33.698 -> 
11:18:33.698 -> Backtrace:0x4008b32d:0x3ffb26b00x4008d02d:0x3ffb26f0 0x4008d14f:0x3ffb2710 0x4008d39f:0x3ffb2730 0x400dae9e:0x3ffb2750 0x400d1106:0x3ffb2790 0x400d128e:0x3ffb27d0 0x400d2642:0x3ffb2820 
11:18:33.698 -> 
11:18:33.698 -> 
11:18:33.698 -> 
11:18:33.698 -> 
11:18:33.698 -> ELF file SHA256: 0000000000000000
11:18:33.698 -> 
11:18:33.698 -> Rebooting...

with ->

const int size_tablo = 150000;
EXT_RAM_ATTR int test_tablo_int[size_tablo];

it does not compile ... 150000 is too large for the DRAM the output is ->

.../appdata/local/arduino15/packages/esp32/tools/xtensa-esp32-elf-gcc/gcc8_4_0-esp-2021r2/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: C:\Users\OLIVIE~1\AppData\Local\Temp\arduino_build_803706/sketch_feb15a.ino.elf section `.dram0.bss' will not fit in region `dram0_0_seg'
c:/users/olivier_pailler/appdata/local/arduino15/packages/esp32/tools/xtensa-esp32-elf-gcc/gcc8_4_0-esp-2021r2/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: DRAM segment data does not fit.
c:/users/olivier_pailler/appdata/local/arduino15/packages/esp32/tools/xtensa-esp32-elf-gcc/gcc8_4_0-esp-2021r2/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: DRAM segment data does not fit.
c:/users/olivier_pailler/appdata/local/arduino15/packages/esp32/tools/xtensa-esp32-elf-gcc/gcc8_4_0-esp-2021r2/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: region `dram0_0_seg' overflowed by 491792 bytes
collect2.exe: error: ld returned 1 exit status
exit status 1
Erreur de compilation pour la carte ESP32 Dev Module

i hope it's clearer ... mmm not so sure !

maybe it is caused by what said me-no-dev : "Arduino is compiled with the option to not add PSRAM to the general heap,"

i noticed comparing builds with with WinMerge that only libesp_system.a and libapp_update.a were different ... and also libarduino.a ...

it looks like ARDUINO IDE doesnot like this setup -> CONFIG_SPIRAM_BOOT_INIT=y

and if it is not set, we cannot use EXT_RAM_ATTR for a global variable (for both ARDUINO IDE and ECLIPSE ESP-IDE)

ouf ... olive

me-no-dev commented 2 years ago

maybe the reason is the fact that Arduino initializes PSRAM a bit later than ESP-IDF. This is because we use prebuilt IDF libs with PSRAM support, but in most cases PSRAM is not available and needs to be enabled by the user in the board menu of Arduino IDE. Since v2.0.0, we also add PSRAM to malloc, but as I said... happens slightly later. You can define the array as pointer and allocate it in setup(). This will fix your issue.

sonolive commented 2 years ago

hello me-no-dev thanx alot for this answer,

maybe the reason is the fact that Arduino initializes PSRAM a bit later than ESP-IDF

oh yes i didnot think of that bur for sure it could be the problem ...

Since v2.0.0, we also add PSRAM to malloc, but as I said... happens slightly later. You can define the array as pointer and allocate it in setup()

Yes , ok i know that, and it could fix this little sketch ! the fact is that i am looking for a solution for a MUCH LARGER app, based on LVGL gfx library and after few weeks of dev i encounter this issue ... 320.000 KB is not enough, i need much more, and they are here ... 4MB but i cannot use them

realloc dynamically everything should be a real pain ... but maybe i will have to do that ! booooooh

anyway thx you 2 for the job you do for arduino-esp32 regards, olive

star297 commented 1 year ago

If anyone gets here, I had a similar issue, here's a solution. Works on PlatformIO, haven't tried on regular Arduino IDE.

https://community.platformio.org/t/how-to-use-a-global-struct-array-in-psram-esp32-wrover/30696

rtprof commented 1 year ago

If anyone gets here, I had a similar issue, here's a solution. Works on PlatformIO, haven't tried on regular Arduino IDE.

https://community.platformio.org/t/how-to-use-a-global-struct-array-in-psram-esp32-wrover/30696

Good job. Just now I'm searching the solution and here it is!!! Thank you. I tried on Arduino IDE and it works. Now I think how to send all my String variables to PSRAM

davepl commented 9 months ago

Am I correct, then, that it's not possible to allocate BSS and DATA on PSRAM because CONFIG_SPIRAM is hard-coded in the Arduino build?

Or is there a way to get static allocations into PSRAM on an Arduino project?

espressif / arduino-esp32

PSRAM enabled, but not usable as internal heap memory #2226