PSRAM is unusable due to broken ESP-IDF builder

[rohansingh]

According to documentation, adding some build flags is sufficient to enable PSRAM support and enable the PSRAM/SPI RAM cache workaround. For background, ESP32 chips have a silicon bug that causes memory inconsistency unless this workaround is enabled at compile & link time.

The documentation says to enable the workaround with the -DCONFIG_SPIRAM_CACHE_WORKAROUND build flag. However, this flag doesn't actually do anything. The flag itself is handled by the ESP-IDF build system, which PlatformIO doesn't use. So it has no actual effect.

Trying to use PSRAM results in memory inconsistencies and crashes. Random bytes in PSRAM will flip to zero and other weird stuff will happen. Overall, the system is completely unstable. This may also be an issue for Arduino users, but I've only tested with ESP-IDF.

To fix this problem, the build system needs to do a few different things when CONFIG_SPIRAM_CACHE_WORKAROUND is enabled:

1. Pass the -mfix-esp32-psram-cache-issue flag to the compiler to ensure only compatible code is emitted.
2. Ensure that libc-psram-workaround.a and libm-psram-workaround.a are linked instead of the default libc.a and libm.a.
3. Skip esp32.rom.spiram_incompatible_fns.ld during linking, since it is explicitly SPI RAM-incompatible. Instead, pass esp32-spiram-rom-functions.lf to ldgen.py.

Also, ldgen.py needs to be pointed to a valid sdkconfig file. Currently, generate_project_ld_script points it to sdkconfig.h, which it cannot process.

I've addressed these issues in a fork. I'm happy to open a pull request if this looks acceptable: https://github.com/tidbyt/platform-espressif32/commit/69359a84e88c4d603ab202ef0f884f087ae93805

Finally, if you want PSRAM to work, you need these fixes and also need, at minimum, the following config in sdkconfig:

CONFIG_SPIRAM_SUPPORT=y
CONFIG_SPIRAM_CACHE_WORKAROUND=y

// either of the following will work
CONFIG_SPIRAM_USE_MALLOC=y
CONFIG_SPIRAM_USE_CAPS_ALLOC=y

I think #175 tries to address this, but I'm not sure if build flags are sufficient, since ldgen actually looks at the sdkconfig file directly.

[rohansingh]

Just for posterity, this was originally reported as GoogleCloudPlatform/iot-device-sdk-embedded-c#90, and then in espressif/esp-idf#2892, before it became clear that the issue was with the PlatformIO build system.

[valeros]

Hi @rohansingh ! Starting with ESP-IDF v4.0 we use the native configuration system. Could you please try your project with the upstream version of the platform, for example:

[env:esp32dev]
platform = https://github.com/platformio/platform-espressif32.git
board = esp32dev
framework = espidf

Thanks!

[davidjade]

Hi @valeros I can give a very simple scenario that shows that this is still broken in PIO 4.3.1 and Espressif 32 1.12.0 but I cannot figure out why. It seems like it is building correctly but something is different vs. building the same project with ESP-IDF directly.

Here's a very simple scenario where if you enable the PSRAM option on an ESP32 project (using the new support for ESP-IDF 4.0 & menuconfig), it causes a kernel fault during boot. I am using the esp-wrover-kit board. This is a scenario that works just fine if I take the exact same source files and use ESP-IDF directly to build the project. I am being careful to use the same framework versions and tools for both scenarios.

Here's how to recreate the issue:

1. Start with the ESP32 hello-world example from the ESP-IDF SDK
2. Use the menuconfig to enable the ESP32 option for PSRAM (Component Config, ESP32-specific, check the "Support for external SPI-connected RAM option")
3. Compile, flash, then monitor

If I do just these steps using ESP-IDF 4.0 directly, the hello-world example runs without errors. If I do those same steps in PlatformIO as an espidf project, then the board crashes early in the boot process with a stream of faults about illegal instructions:

(218) spiram: Found 32MBit SPI RAM device
(222) spiram: SPI RAM mode: flash 40m sram 40m
(227) spiram: PSRAM initialized, cache is in low/high (2-core) mode.
(234) cpu_start: Pro cpu up.
(238) cpu_start: Application information:
(243) cpu_start: Project name:     esp32psram
(248) cpu_start: App version:      1
(252) cpu_start: Compile time:     Mar 26 2020 01:16:08
(258) cpu_start: ELF file SHA256:  ef5217cd13489a40...
(264) cpu_start: ESP-IDF:          HEAD-HASH-NOTFOUND
(270) cpu_start: Starting app cpu, entry point is 0x40082aec
(0) cpu_start: App cpu up.
(1172) spiram: SPI SRAM memory test OK
(1172) heap_init: Initializing. RAM available for dynamic allocation:
(1172) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
(1178) heap_init: At 3FFB31D0 len 0002CE30 (179 KiB): DRAM
(1185) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
(1191) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
(1198) heap_init: At 4008C634 len 000139CC (78 KiB): IRAM
(1204) cpu_start: Pro cpu start user code
(1209) spiram: Adding pool of 4096K of external SPI memory to heap allocator
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x400e9084: bad00bad bad00bad bad00bad
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x400e9084: bad00bad bad00bad bad00bad
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x400e9084: bad00bad bad00bad bad00bad

Note that the only change to the SDK example program is using menuconfig to turn on PSRAM. It should be the same change in both projects and in fact the resulting sdkconfig that menuconfig creates file is identical in both of the projects.

I've scoured the build output files for both projects looking for a reason and I can't find anything obvious. I verified that the same versions of the xtensa compiler, libs, and ESP-IDF frameworks where being used in both cases (in case there was any difference). I have greped for the important flags mentioned in https://github.com/platformio/platform-espressif32/issues/254#issue-533689521

[valeros]

Hi @davidjade ! Many thanks for the report. I've created a special branch with some fixes for projects with enabled PSRAM, could you please try it with yours?

[env:esp32dev]
platform = https://github.com/platformio/platform-espressif32.git#feature/espidf-psram-support
board = esp32dev
framework = espidf

[davidjade]

Hi @valeros Success!

btw, I use these two lines added to the example to verify that the PSRAM is available to the heap. I have not checked functionality beyond this.

printf("Free heap size: %d\n", (int) xPortGetFreeHeapSize());
heap_caps_print_heap_info(MALLOC_CAP_DEFAULT);

So I am curious, how do your changes work? Looking at the changes, I see stuff that looks like the old PIO fix, swapping out the broken newlib ROM stuff (but apparently not using the PSRAM libc & libm replacements?).

I didn't expect to see this type of fix as needed as I thought that the new PIO was just using the ESP-IDF CMake system and the ESP-IDF build system would take care of everything? So what's the interaction between the two systems now? Is it some sort of hybrid? I'd like to get a better understanding of this.

And what about the libc-psram-workaround.a & libm-psram-workaround.a changes? Have these been depreciated in ESP-IDF 4.0? I had thought they would still be needed but I can't seem to figure it out. Where do they come into play if still needed?

I am thrilled btw, to finally have this working. It has stopped me from using PlatformIO in the past as all of my boards have PSRAM.

[valeros]

@davidjade

So I am curious, how do your changes work? Looking at the changes, I see stuff that looks like the old PIO fix, swapping out the broken newlib ROM stuff (but apparently not using the PSRAM libc & libm replacements?).

I didn't expect to see this type of fix as needed as I thought that the new PIO was just using the ESP-IDF CMake system and the ESP-IDF build system would take care of everything? So what's the interaction between the two systems now? Is it some sort of hybrid? I'd like to get a better understanding of this.

Yes, it's somewhat hybrid, but not as much as it was with v3.x. In a nutshell, CMake doesn't export (at least in the latest stable release) actual commands for targets added via the add_custom_target function, so we need to reproduce these targets in our build script manually. One of these custom targets is the command for generating the project linker script and the problem was that in our build script this command simply lacked an additional linker script fragment which takes into account the PSRAM (that's why I need the actual SDK configuration to figure out whether the PSRAM is enabled).

And what about the libc-psram-workaround.a & libm-psram-workaround.a changes? Have these been depreciated in ESP-IDF 4.0? I had thought they would still be needed but I can't seem to figure it out. Where do they come into play if still needed?

Looks like this has been fixed in the latest toolchain:

if(GCC_NOT_5_2_0)
    if(CONFIG_NEWLIB_NANO_FORMAT)
        set(LIBC c_nano)
    else()
        set(LIBC c)
    endif()

    set(LIBM m)
else()
    if(CONFIG_SPIRAM_CACHE_WORKAROUND)
        set(LIBC c-psram-workaround)
        set(LIBM m-psram-workaround)
    else()
        if(CONFIG_NEWLIB_NANO_FORMAT)
            set(LIBC c_nano)
        else()
            set(LIBC c)
        endif()

        set(LIBM m)
    endif()
endif()

[davidjade]

Thanks for the explanation and pointers - this helps.

From my reading of the various cmake files, it seems that when using GCC 8.2.0 now, these special libs are no longer being linked in. It seems that the way used to swap out the newlib ROM functions when using PSRAM has changed in ESP-IDF?

If that's the case, then hopefully this discussion can help anyone else that sees this as there seems to be a lot of confusion on what steps are needed for PSRAM to work. Both good and bad (for those trying the old ways) that it has changed and maybe gotten simpler.