External PSRAM not supported?

[jgamble-simple]

Hardware:

Board: Custom board based on ESP32-WROVER-E Core Installation version: 1.0.6 IDE name: Platform.io PSRAM enabled: yes Upload Speed: 230400 Computer OS: Windows 10 (Using wsl for uploading/platform.io)

Description:

Calling heap_caps_malloc_extmem_enable(0) causes a crash about 20% of the time when trying to connect to wifi, I have trimmed back my application to the bare min required to crash it. If I remove the heap_caps_malloc_extmem_enable call then it works properly........

Is this supposed to work? Or this crash expected?

Compiler Flags defined in platformio.ini:

build_unflags = -Os
build_flags = 
   -DCORE_DEBUG_LEVEL=5
   -DBOARD_HAS_PSRAM=1
   -DCONFIG_SPIRAM_CACHE_WORKAROUND=1
   -mfix-esp32-psram-cache-issue
   -O3 -ffast-math
   -Wl,-Map,output.map

    -DCONFIG_ARDUINO_ISR_IRAM=1

Sketch:

#include <WiFi.h>
#include "esp_wifi.h"
#include "esp_heap_caps.h"

void setup() {
  heap_caps_malloc_extmem_enable(0);

  WiFi.begin("", ""); // Credentials removed
}

void loop() {
}

Debug Messages:

ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:10124
load:0x40080400,len:5828
entry 0x400806a8
[D][esp32-hal-psram.c:47] psramInit(): PSRAM enabled
[D][WiFiGeneric.cpp:374] [D][WiFiGeneric.cpp:374] _eventCallback(): Event: 2 - STA_START
_eventCallback(): Event: 0 - WIFI_READY
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Core 0 register dump:
PC      : 0x4008d508  PS      : 0x00060034  A0      : 0x8008e14c  A1      : 0x3ffbf790
A2      : 0x3f800de4  A3      : 0xb33fffff  A4      : 0x3ffbf7cc  A5      : 0xbad00bad
A6      : 0x00000001  A7      : 0x00000001  A8      : 0x00000000  A9      : 0x00000000
A10     : 0x00000000  A11     : 0x00000000  A12     : 0x40062229  A13     : 0x3ffbf8b0
A14     : 0x00e0a000  A15     : 0x3ffbfec8  SAR     : 0x00000018  EXCCAUSE: 0x00000007
EXCVADDR: 0x00000000  LBEG    : 0x4008b190  LEND    : 0x4008b1ac  LCOUNT  : 0x00000000
Core 0 was running in ISR context:
EPC1    : 0x40062229  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x4008d508

ELF file SHA256: 0000000000000000

Backtrace: 0x4008d508:0x3ffbf790 0x4008e149:0x3ffbf7c0 0x4008cd95:0x3ffbf7f0 0x40083bd6:0x3ffbf810 0x40081522:0x3ffbf830 0x4008143f:0x3ffbf860 0x40081917:0x3ffbf880 0x400891c5:0x3ffbf8b0 0x40062226:0x3ffb3f80 0x4009243f:0x3ffb3fa0 0x4009247a:0x3ffb3fd0 0x40092506:0x3ffb4000 0x40092975:0x3ffb4020 0x40089c78:0x3ffb4040 0x40089d0a:0x3ffb4060 0x401353f7:0x3ffb4080 0x4013577d:0x3ffb40a0 0x401335ea:0x3ffb40c0 0x40133b02:0x3ffb4140 0x40132f07:0x3ffb41b0 0x400df10a:0x3ffb41f0 0x400dd158:0x3ffb4210 0x400eb689:0x3ffb4230 0x400de76b:0x3ffb42b0 0x4010a497:0x3ffb42d0 0x401086aa:0x3ffb42f0 0x401097cc:0x3ffb4310 0x400fe3e7:0x3ffb43a0 0x40100666:0x3ffb43d0 0x4008d1ca:0x3ffb4400
  #0  0x4008d508:0x3ffbf790 in uxPortCompareSetExtram at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:452
  #1  0x4008e149:0x3ffbf7c0 in vPortCPUAcquireMutexIntsDisabledExtram at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:3507
      (inlined by) vPortCPUAcquireMutexIntsDisabled at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portmux_impl.h:95
      (inlined by) vTaskEnterCritical at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4201
  #2  0x4008cd95:0x3ffbf7f0 in xQueueGenericSendFromISR at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:2038
  #3  0x40083bd6:0x3ffbf810 in queue_send_from_isr_wrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/esp_adapter.c:324
  #4  0x40081522:0x3ffbf830 in pp_post at ??:?
  #5  0x4008143f:0x3ffbf860 in lmacProcessRxSucData at ??:?
  #6  0x40081917:0x3ffbf880 in wDev_ProcessFiq at ??:?
  #7  0x400891c5:0x3ffbf8b0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
  #8  0x40062226:0x3ffb3f80 in ?? ??:0
  #9  0x4009243f:0x3ffb3fa0 in esp_rom_spiflash_read_status at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_rom_patch.c:662
  #10 0x4009247a:0x3ffb3fd0 in esp_rom_spiflash_wait_idle at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_rom_patch.c:662
  #11 0x40092506:0x3ffb4000 in esp_rom_spiflash_erase_sector_internal at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_rom_patch.c:662
  #12 0x40092975:0x3ffb4020 in esp_rom_spiflash_erase_sector at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_rom_patch.c:662
  #13 0x40089c78:0x3ffb4040 in spi_flash_erase_range at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/flash_ops.c:247
  #14 0x40089d0a:0x3ffb4060 in spi_flash_erase_sector at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/flash_ops.c:215
  #15 0x401353f7:0x3ffb4080 in nvs::Page::erase() at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_page.cpp:876
  #16 0x4013577d:0x3ffb40a0 in nvs::PageManager::requestNewPage() at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_pagemanager.cpp:184
  #17 0x401335ea:0x3ffb40c0 in nvs::Storage::writeMultiPageBlob(unsigned char, char const*, void const*, unsigned int, nvs::VerOffset) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_storage.cpp:579
  #18 0x40133b02:0x3ffb4140 in nvs::Storage::writeItem(unsigned char, nvs::ItemType, char const*, void const*, unsigned int) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_storage.cpp:579
  #19 0x40132f07:0x3ffb41b0 in nvs_set_blob at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_api.cpp:547
  #20 0x400df10a:0x3ffb41f0 in wifi_nvs_set at ??:?
  #21 0x400dd158:0x3ffb4210 in wifi_station_save_ap_channel at ??:?
  #22 0x400eb689:0x3ffb4230 in cnx_auth_done at ??:?
  #23 0x400de76b:0x3ffb42b0 in ieee80211_auth_done at ??:?
  #24 0x4010a497:0x3ffb42d0 in wpa_neg_complete at ??:?
  #25 0x401086aa:0x3ffb42f0 in wpa_parse_kde_ies at ??:?
  #26 0x401097cc:0x3ffb4310 in eapol_txcb at ??:?
  #27 0x400fe3e7:0x3ffb43a0 in ppProcTxDone at ??:?
  #28 0x40100666:0x3ffb43d0 in ppTask at ??:?
  #29 0x4008d1ca:0x3ffb4400 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

Rebooting...
ets Jul 29 2019 12:21:46

[igrr]

I think the root cause is that behavior of some IDF drivers depends on CONIFG_SPIRAM_USE_MALLOC Kconfig option. For example, if this option is enabled, Wi-Fi driver will use a slightly more complex code to allocate queues in internal RAM: https://github.com/espressif/esp-idf/blob/3e370c4296247b349aa3b9a0076c05b9946d47dc/components/esp_wifi/esp32/esp_adapter.c#L125-L153. The authors of that code assumed that if CONFIG_SPIRAM_USE_MALLOC is disabled, it means that malloc only returns memory from internal RAM.

I think the solution for this problem is to enable CONIFG_SPIRAM_USE_MALLOC by default — which is done in arduino-esp32 2.0.0 and later releases. There are some downsides to that, like increased heap usage, but there is work in progress to fix this issue.

[VojtechBartoska]

@SuGlider PTAL on this related issue.

[SuGlider]

@jgamble-simple

Unfortunatelly 2.0.0 is not availble to PlaformIO at this moment. But it can be used with Arduino IDE.

I created a repository with a new Arduino Core 2.0.0 that can be tested. It is part of modifications related to heap and binary allocation (#5630), and it also has CONIFG_SPIRAM_USE_MALLOC enabled.

https://github.com/espressif/arduino-esp32/tree/mem-optimized

In order to install it as a separated board for testing, please follow the instructions from https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html The only difference here would be to clone mem-optimized branch instead.

git clone -b mem-optimized https://github.com/espressif/arduino-esp32 esp32

In case it works fine for a number of users, we will commit it for the next ESP32 Arduino release. Please test it with your project and let me know.

[jgamble-simple]

@SuGlider

Actually, I figured out how to get arduino version 2.0.0 for the esp-idf by modifying my ini file:

platform = https://github.com/platformio/platform-espressif32.git#feature/arduino-upstream
platform_packages =
  framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.0
board = esp-wrover-kit
framework = arduino

So far it works with a couple tweaks to the existing code.....

[mrdc]

So far it works

@igrr @jgamble-simple

Hi!

I've tried your solution and heap haven't increased compared to pre-2.0.0 version :( CONFIG_SPIRAM_USE_MALLOC=y is in sdkconfig by default in 2.0.0+, but getFreeHeap shows 30k free after boot. Even less than with 1.0.5 version, which showed 65k free for the same firmware.

[jgamble-simple]

You still have to call heap_caps_malloc_extmem_enable(0); which will enable the external heap for a malloc call.

The default is to make all malloc calls use internal heap. The argument is the smallest allocation that would still be allocated on internal heap.

Also I am not sure what the getFreeHeap function returns, so I am not sure if that is supposed to reflect a larger heap.

[SuGlider]

@jgamble-simple getFreeHeap returns actually free DRAM and IRAM. Only DRAM accounts for heap. In order to get an accurate free heap amount, please use heap_caps_get_free_size(MALLOC_CAP_8BIT)

For more information: hhttps://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/mem_alloc.html#memory-capabilities

[mrdc]

In order to get an accurate free heap amount, please use heap_caps_get_free_size(MALLOC_CAP_8BIT)

heap_caps_get_free_size(MALLOC_CAP_8BIT) shows 4394040.

It looks like ssl_client.cpp (WiFiClientSecure) uses getFreeHeap (getHeapSize shows 219044) :

[ssl_client.cpp:56] start_ssl_client(): Free internal heap before TLS 177812
...
[ssl_client.cpp:240] start_ssl_client(): Free internal heap after TLS 138528

and it fails when FreeHeap is low. Does it mean that WiFiClientSecure (from arduino-esp32 2.0) uses only internal RAM and not PSRAM?

[lbernstone]

and it fails when FreeHeap is low. Does it mean that WiFiClientSecure (from arduino-esp32 2.0) uses only iternal RAM and not PSRAM?

WiFiClientSecure is using the hardware accelerated encryption functions. AFAIK, there is no way to use the PSRAM to feed/collect the hardware accelerator. There is a way to use DMA with the HW AES, so it might be possible to keep this off the heap, but that would be beyond my skills, and would likely need some changes in IDF.

[igrr]

Note that mbedTLS memory allocation policy can be configured, currently it is fixed to "internal RAM": https://github.com/espressif/esp32-arduino-lib-builder/blob/51a3ba2bcd1eaad61ff9d67de5cc93b5baeeb9f8/sdkconfig.esp32#L1152-L1155

[lbernstone]

Very interesting. I may have to play around with that at some point. https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/kconfig.html#config-mbedtls-mem-alloc-mode

[mrdc]

Note that mbedTLS memory allocation policy can be configured, currently it is fixed to "internal RAM"

I've just checked it (framework-arduinoespressif32 0.0.0+sha.44c1198): CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y

# CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y
# CONFIG_MBEDTLS_DEFAULT_MEM_ALLOC is not set
# CONFIG_MBEDTLS_CUSTOM_MEM_ALLOC is not set
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=16384
# CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN is not set
# CONFIG_MBEDTLS_DYNAMIC_BUFFER is not set
# CONFIG_MBEDTLS_DEBUG is not set

[ 26130][V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 171639
...
[ 27635][V][ssl_client.cpp:294] start_ssl_client(): Free internal heap after TLS 132607

and CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y

CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y
# CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y
# CONFIG_MBEDTLS_DEFAULT_MEM_ALLOC is not set
# CONFIG_MBEDTLS_CUSTOM_MEM_ALLOC is not set
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=16384
# CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN is not set
# CONFIG_MBEDTLS_DYNAMIC_BUFFER is not set
# CONFIG_MBEDTLS_DEBUG is not set

[ 26141][V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 171639
...
[ 27642][V][ssl_client.cpp:294] start_ssl_client(): Free internal heap after TLS 132607

Gives the same heap size...

[igrr]

@mrdc Could you please clarify where have you specified CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y? Did you rebuild arduino-esp32 libraries using arduino-lib-builder?

[mrdc]

Could you please clarify where have you specified CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y?

Sure, here you go:\.platformio\packages\framework-arduinoespressif32@src-2567d2800cf655927e3f141014aee93c\tools\sdk\esp32\sdkconfig

Did you rebuild arduino-esp32 libraries using arduino-lib-builder?

Not yet, have to check first how to do it :)

[SuGlider]

@mrdc Changes to sdkconfig only take effect when the libraries are rebuilt and copied using the scripts from https://github.com/espressif/esp32-arduino-lib-builder

Please check the latest Arduino release v2.0.1RC1 that makes more heap sapce available, reduces the binary size and keeps CONIFG_SPIRAM_USE_MALLOC enabled.

https://github.com/espressif/arduino-esp32/releases/

[iwas108]

Any update on this?

[raphael-bmec-co]

I am seeing what I think is the same issue on Arduino Core 2.0.1 AND 2.0.3-rc1.

Is anyone else? How sure are we that this is resolved?

Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed).

Core 0 register dump: PC : 0x40085382 PS : 0x00060c35 A0 : 0x80085454 A1 : 0x3ffbeb0c A2 : 0x3f8291cc A3 : 0xbad00bad A4 : 0x00060a23 A5 : 0x40094d7c A6 : 0x003fffff A7 : 0x3ffdb0dc A8 : 0x3ffbcfe0 A9 : 0x3ffbebfc A10 : 0x00000011 A11 : 0x00000000 A12 : 0x0000186a A13 : 0x00000000 A14 : 0x0000186a A15 : 0x00004000 SAR : 0x0000000f EXCCAUSE: 0x00000007 EXCVADDR: 0x00000000 LBEG : 0x400917e4 LEND : 0x400917fa LCOUNT : 0x00000000

Backtrace:0x4008537f:0x3ffbeb0c |<-CORRUPTED

0 0x4008537f:0x3ffbeb0c in esp_timer_stop at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/e

sp-idf/components/esp_timer/src/esp_timer.c:192

ELF file SHA256: 0000000000000000

[jgamble-simple]

I am 100% sure that it works on 2.0.0.... But I am not sure that it works on later versions, because I am still on 2.0.0 for my project @raphael-bmec-co

[raphael-bmec-co]

@jgamble-simple thanks for this. I will role back to 2.0.0 and see if I am still seeing the crash. Appreciate the prompt feedback.

[raphael-bmec-co]

Okay I have done days and days of testing on this and I am afraid it is 100% not working as detailed below.

This seems like a huge stability issue to me. Using BLE+WiFi+PSRAM is a base use case for secure IoT applications. Hopefully this is something that is resolvable. The fact that it was working on 1.0.6 is promising.

Can we reopen this issue or should I copy this across to a new issue?

Issue details

Board: ESP32 Dev Module

Device: Chip is ESP32-D0WD-V3 (revision 3); Auto-detected Flash size: 16MB

Hardware Configuration: None

Version: 1.0.6; 2.0.0; 2.0.1; 2.0.2; 2.0.3-RC1

IDE Name: Arduino IDE AND CLion with PIO

OS: Windows 10

Flash frequency: 80 MHz

PSRAM enabled: Yes

Upload speed: 921600

Description Using BLE + WiFi + heap_caps_malloc_extmem_enable(0) causes a spontaneous crash when the AP is not found and also at other random times. The AP not found case is most easily replicated and is detailed below:

1.0.6: NO CRASH

2.0.0: Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed).

Stack trace and decode

> Core 0 register dump: > PC : 0x400855cd PS : 0x00060e34 A0 : 0x80085714 A1 : 0x3ffbe960 > A2 : 0x3f813b40 A3 : 0xbad00bad A4 : 0x00060a23 A5 : 0x40086204 > A6 : 0x3ffc3f68 A7 : 0x00000017 A8 : 0x00000103 A9 : 0xbad00bad > A10 : 0x00000010 A11 : 0x00000000 A12 : 0x0000186a A13 : 0x0000f604 > A14 : 0xfffff000 A15 : 0x00000008 SAR : 0x00000010 EXCCAUSE: 0x00000007 > EXCVADDR: 0x00000000 LBEG : 0x40091328 LEND : 0x40091344 LCOUNT : 0x00000000 > > > Backtrace:0x400855ca:0x3ffbe9600x40085711:0x3ffbe980 0x40083671:0x3ffbe9a0 0x4008dfa3:0x3ffbe9c0 0x4008d7cf:0x3ffbe9e0 0x4008d8f3:0x3ffbea00 0x4008d5e2:0x3ffbea30 0x4008620d:0x3ffbea50 0x40089ded:0x3ffbea70 0x40086bce:0x3ffbeab0 0x4008c3ae:0x3ffbead0 0x4008cec7:0x3ffbeaf0 0x40085ca5:0x3ffbeb10 0x400914b3:0x3ffbc550 0x4009a44c:0x3ffbc560 0x40081fd5:0x3ffbc5e0 0x400e167e:0x3ffbc620 0x400e35fe:0x3ffbc650 0x400e4129:0x3ffbc670 0x400e44eb:0x3ffbc690 0x400e31b1:0x3ffbc710 0x400e359f:0x3ffbc780 0x400e1bc6:0x3ffbc7a0 0x40101b19:0x3ffbc7c0 0x400e999f:0x3ffbc7f0 0x400e93fe:0x3ffbc810 0x400efe09:0x3ffbc830 > PC: 0x400855cd: esp_timer_stop at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer.c line 308 > EXCVADDR: 0x00000000 > > Decoding stack results > 0x400855ca: esp_timer_stop at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer.c line 308 > 0x40085711: ets_timer_disarm at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/ets_timer_legacy.c line 109 > 0x40083671: timer_disarm_wrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_wifi/esp32/esp_adapter.c line 438 > 0x4008620d: coex_bt_request_wrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/bt/controller/esp32/bt.c line 1037 > 0x4009a44c: spi_flash_chip_generic_read at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_chip_generic.c line 213 > 0x40081fd5: esp_flash_read at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/esp_flash_api.c line 712 > 0x400e167e: esp_partition_read at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/partition.c line 415 > 0x400e35fe: nvs::NVSPartition::read(unsigned int, void*, unsigned int) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_partition.cpp line 45 > 0x400e4129: nvs::Page::readEntry(unsigned int, nvs::Item&) const at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_page.hpp line 209 > 0x400e44eb: nvs::Page::findItem(unsigned char, nvs::ItemType, char const*, unsigned int&, nvs::Item&, unsigned char, nvs::VerOffset) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_page.cpp line 841 > 0x400e31b1: nvs::Storage::eraseItem(unsigned char, nvs::ItemType, char const*) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_storage.cpp line 160 > 0x400e359f: nvs::NVSHandleSimple::erase_item(char const*) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_storage.hpp line 98 > 0x400e1bc6: nvs_erase_key(nvs_handle_t, char const*) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_api.cpp line 335 > 0x40101b19: dhcp_ip_addr_erase at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/port/esp32/netif/dhcp_state.c line 74 > 0x400e999f: esp_netif_dhcpc_stop_api at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_netif/lwip/esp_netif_lwip.c line 1042 > 0x400e93fe: esp_netif_api_cb at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_netif/lwip/esp_netif_lwip.c line 122 > 0x400efe09: tcpip_thread at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c line 208

2.0.1: Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed).

Stack trace and decode

> Core 0 register dump: > PC : 0x40085470 PS : 0x00060535 A0 : 0x800855a0 A1 : 0x3ffbe910 > A2 : 0x3f80d8e4 A3 : 0xbad00bad A4 : 0x00060523 A5 : 0xb33fffff > A6 : 0x00000001 A7 : 0x00000001 A8 : 0xbad00bad A9 : 0x3ffbe8e0 > A10 : 0x00000003 A11 : 0x00060523 A12 : 0x00060523 A13 : 0x40094068 > A14 : 0x00000012 A15 : 0x3ffbcbdc SAR : 0x0000000f EXCCAUSE: 0x00000007 > EXCVADDR: 0x00000000 LBEG : 0x4009183c LEND : 0x40091858 LCOUNT : 0x00000000 > > > Backtrace:0x4008546d:0x3ffbe9100x4008559d:0x3ffbe930 0x40083829:0x3ffbe950 0x4008e3ff:0x3ffbe970 0x4008ddf1:0x3ffbe990 0x4008da9d:0x3ffbe9c0 0x4008605d:0x3ffbe9e0 0x40086396:0x3ffbea00 0x40085c75:0x3ffbea30 0x4008d20b:0x3ffbeaf0 0x40085b25:0x3ffbeb10 0x4009750e:0x3ffbc980 0x40099d3d:0x3ffbc9b0 0x40099e04:0x3ffbc9d0 0x40082216:0x3ffbca50 0x400df5f0:0x3ffbca90 0x400e0f3a:0x3ffbcac0 0x400e14c9:0x3ffbcae0 0x400e1ced:0x3ffbcb00 0x400dff46:0x3ffbcb70 0x400e0835:0x3ffbcbc0 0x400e0ecd:0x3ffbcc20 0x400df9fb:0x3ffbcc40 0x400f9db5:0x3ffbcc70 0x400e68ba:0x3ffbcca0 0x400e6246:0x3ffbccc0 0x400eb89b:0x3ffbcce0 >Decode Failed (Arduino IDE)

2.0.2: Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed).

Stack trace and decode

Core 0 register dump: > PC : 0x400854a0 PS : 0x00060235 A0 : 0x800855d0 A1 : 0x3ffbe9cc > A2 : 0x3f80d8b4 A3 : 0xbad00bad A4 : 0x00060023 A5 : 0x400940b0 > A6 : 0x007bc200 A7 : 0x003fffff A8 : 0xbad00bad A9 : 0x3ffbeabc > A10 : 0x00000011 A11 : 0x00000000 A12 : 0x0000186a A13 : 0x00000000 > A14 : 0x0000186a A15 : 0x3ffd425c SAR : 0x0000000f EXCCAUSE: 0x00000007 > EXCVADDR: 0x00000000 LBEG : 0x40091874 LEND : 0x40091890 LCOUNT : 0x00000000 > > > Backtrace:0x4008549d:0x3ffbe9cc |<-CORRUPTED >Decode Failed (Arduino IDE)

2.0.3-RC1: Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed).

Stack trace and decode

Core 0 register dump: > PC : 0x400848cc PS : 0x00060535 A0 : 0x8008499c A1 : 0x3ffbea8c > A2 : 0x3f80dad0 A3 : 0xbad00bad A4 : 0x00060523 A5 : 0x0000cdcd > A6 : 0x003fffff A7 : 0x00060523 A8 : 0xbad00bad A9 : 0x3ffbea5c > A10 : 0x00000003 A11 : 0x00060523 A12 : 0x00060523 A13 : 0x400929f4 > A14 : 0x003fffff A15 : 0xa60003c9 SAR : 0x0000000f EXCCAUSE: 0x00000007 > EXCVADDR: 0x00000000 LBEG : 0x4009078c LEND : 0x400907a2 LCOUNT : 0x00000000 > > > Backtrace:0x400848c9:0x3ffbea8c |<-CORRUPTED > #0 0x400848c9:0x3ffbea8c in esp_timer_stop at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer.c:192

Sketch

#include <WiFi.h>
#include <BLEDevice.h>
#include <BLEUtils.h>
#include <BLEServer.h>

// Set to 1 to accelerate the crash. Set to zero to see a typical use case.
#define ACCELERATED_CRASH 1

// BLE defines.
#define SERVICE_UUID        "4fafc201-1fb5-459e-8fcc-c5c9c331914b"
#define CHARACTERISTIC_UUID "beb5483e-36e1-4688-b7f5-ea07361b26a8"

void setup() {
  // Comment this line and the crash does not happen.
  heap_caps_malloc_extmem_enable(0);

  // Serial.
  Serial.begin(115200);
  Serial.println("Starting BLE...");

  // Minimum BLE simplified from example.
  BLEDevice::init("Long name works now");
  BLEServer *pServer = BLEDevice::createServer();
  BLEService *pService = pServer->createService(SERVICE_UUID);
  BLECharacteristic *pCharacteristic = pService->createCharacteristic(
                                         CHARACTERISTIC_UUID,
                                         BLECharacteristic::PROPERTY_READ |
                                         BLECharacteristic::PROPERTY_WRITE
                                       );

  pCharacteristic->setValue("Hello World says Neil");
  pService->start();
  BLEAdvertising *pAdvertising = BLEDevice::getAdvertising();
  pAdvertising->addServiceUUID(SERVICE_UUID);
  BLEDevice::startAdvertising();
  Serial.println("Started BLE");

  // Minimum WiFi.
  Serial.println("Starting WiFi connection attempt...");
  WiFi.begin("missing network", "password");

  Serial.print("Available heap:");
  Serial.println(ESP.getFreeHeap());
}

void loop() {

  // This causes a crash to happen faster and may be useful for debugging.
  // However the crash will happen even without this block.
  if(ACCELERATED_CRASH){
    while(!WiFi.isConnected()){
      log_i("Attempting connection...");
      WiFi.begin("missing network", "password");
      delay(100);
    }
  }

  delay(1000);
  Serial.println("Loop");
}

Potentially related issues: https://github.com/espressif/esp-idf/issues/5129 https://github.com/espressif/esp-idf/issues/5214 https://esp32.com/viewtopic.php?t=26875

[raphael-bmec-co]

@VojtechBartoska it seems like this may have it's root cause in the IDF according to these comments: https://github.com/espressif/esp-idf/issues/7249#issuecomment-1103830017

It seems like it may also be causing other issues with other components in the Arduino core.

Are you able to give me some picture of if patches to the IDF are propagated into the Arduino cores and how long this might take?

Thanks!

[VojtechBartoska]

Hello, rough realistic timeline is 1st week of May fixed in IDF and sync to Arduino.

[raphael-bmec-co]

@VojtechBartoska thank you. We will follow-up early may to check on progress. Much appreciated.

[VojtechBartoska]

I'm adding this issue to our Roadmap to keep eyes on it. :)

[raphael-bmec-co]

@VojtechBartoska thank you.

@Jason2866 I'm relying on your PIO packages. Thank you! Once this fix is implemented, will you be able to create a new PIO package?

[Jason2866]

@raphael-bmec-co If the patch is merged in IDF44 and you ping me i will do a build.

[raphael-bmec-co]

@Jason2866 Fantastic. Thank you!

[raphael-bmec-co]

@Jason2866 looks like they patched it a couple of days back: https://github.com/espressif/esp-idf/issues/7249#issuecomment-1110513248

Hopefully this sorts issues for a bunch of people.

Thanks again for doing the build.

[Jason2866]

@raphael-bmec-co You can use with

platform = https://github.com/tasmota/platform-espressif32/releases/download/v2.0.3rc1/platform-espressif32-2.0.3new.zip
platform_packages = https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/826/framework-arduinoespressif32-v4.4_work-b98c8476f2.tar.gz

[raphael-bmec-co]

Thanks @Jason2866

pio run is failing with error: Error: Could not find the package with 'esp32-arduino-lib-builder @ *' requirements for your system 'windows_amd64'

[Jason2866]

Ah, sorry correct is

platform = https://github.com/tasmota/platform-espressif32/releases/download/v2.0.3rc1/platform-espressif32-2.0.3new.zip
platform_packages = framework-arduinoespressif32 @ https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/826/framework-arduinoespressif32-v4.4_work-b98c8476f2.tar.gz

[raphael-bmec-co]

Thanks @Jason2866

This compiles but I'm getting a crash loop:

rst:0x3 (SW_RESET),boot:0x17 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DOUT, clock div:1 load:0x3fff0030,len:344 load:0x40078000,len:13328 load:0x40080400,len:3488 entry 0x400805d0 ets Jul 29 2019 12:21:46

I'll post this for the IDF guys too to see if it is their doing and do some more detailed debugging tomorrow, but please let me know if you have any ideas.

[raphael-bmec-co]

Hi @Jason2866

Okay I can't get anything at all to run. I have also tried several different flash modes and frequencies. Before I go back to the IDF guys, is there anything obviously wrong with the Arduino Core build? Are you able to get anything running? Apologies for adding work to your plate.

[Jason2866]

@raphael-bmec-co Tried latest build. All my devices with PSRAM do work and uses the PSRAM. ESP32, ESP32-S2 and ESP32-S3. We / i do not use PSRAM for "everything", since it is slower. We place only RAM consuming (not time critical) functions there. Imho it is a bad idea at all to place wifi task in PSRAM.

[raphael-bmec-co]

@Jason2866 I'll try a bit harder to see if I can get something running.

I'm assuming you ran your test with PIO and the platform and platform_packages you shared? I've got:

[env:esp32dev]
platform = https://github.com/tasmota/platform-espressif32/releases/download/v2.0.3rc1/platform-espressif32-2.0.3new.zip
platform_packages = framework-arduinoespressif32 @ https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/826/framework-arduinoespressif32-v4.4_work-b98c8476f2.tar.gz
board = esp32dev
framework = arduino
upload_port = COM6

;BOARD OPTIONS
build_flags =
    ; Enable PSRAM.
;   -DBOARD_HAS_PSRAM
;   -mfix-esp32-psram-cache-issue
    ; Set cores.
;   -DARDUINO_RUNNING_CORE=1
;   -DARDUINO_EVENT_RUNNING_CORE=1
;   -DCONFIG_ARDUINO_ISR_IRAM=0
;   -DCONFIG_SPIRAM_USE_MALLOC=1
    ; Set comment level.
    ; None
    ;-DCORE_DEBUG_LEVEL=0
    ; Error
    ;-DCORE_DEBUG_LEVEL=1
    ; Warn
    ;-DCORE_DEBUG_LEVEL=2
    ; Info
    -DCORE_DEBUG_LEVEL=3
    ; Debug
    ;-DCORE_DEBUG_LEVEL=4
    ; Verbose
    ;-DCORE_DEBUG_LEVEL=5

; Set frequency to 240MHz.
board_build.f_cpu = 240000000L
; Set partition scheme to https://github.com/espressif/arduino-esp32/tree/master/tools/partitions.
board_build.partitions = default_8MB.csv

; Set the flash size.
board_build.flash_size = 16MB
; Set flash mode.
board_build.flash_mode = qout
; Set flash frequency to 80MHz.
board_build.f_flash = 80000000L

; Set upload speed.
upload_speed = 2000000

monitor_filters = esp32_exception_decoder
monitor_port = COM6
monitor_speed = 115200

We don't have much choice on the PSRAM front. We need BLE and WiFi and we are using Google IoT core so we need WiFi secure. Without messing with sdkConfig and building our own Arduino Core, we have to put everything in PSRAM to get enough space for SSL. Unless I'm missing something here. If you know something I don't I'm all ears!

Thanks.

[Jason2866]

For reducing resources we do not use mbedtls and NimBLE from Arduino for Tasmota. Using a optimized BearSSL implementation and h2zero NimBLE (using lower resources too). So we are able to connect to AWS IoT (custom implementation) if wanted. In general using Bluetooth is a big resource "taker".

[raphael-bmec-co]

@Jason2866 I have narrowed the issue to the partition table selection. 'default.csv' works. default_16MB.csv and default_8MB.csv do not. I'll keep at it but please let me know if you know what is causing this.

I've seen this issue but I'm using a ESP32-D0WDQ5-V3 (revision 3)

[Jason2866]

We do not use any of the predefined partition schemes for Tasmota. May you try one of these. I have esp32 with 16MB and the do work (one of them was used for the tests above) with using our schemes

[raphael-bmec-co]

@Jason2866 nope that also isn't working. I noted in that esp32_8M.json that you seem to be using Arduino as a component? Is this the case? I am using the core only.

[Jason2866]

No, it is Arduino only used. Have you a example code which is failing? If your project fails, you have "somewhere" a issue. We use the flash space from devices > 4Mb for OTA and filesystem and have not encountered problems. No issues from Tasmota users either ever in github or in our Discord. Stability problems arrive only when BLE is heavily used.

[Jason2866]

May you try flashmode dout and flash frequency 40 Mhz and 80 Mhz cpu clock. I have seen some bad devices...

[raphael-bmec-co]

Okay I have testing on two different devices now one with 8MB flash and one with 16MB. Not working as detailed below. Not sure where to go to from here.

This is the trimmed down PIO file showing with partition files work and which don't:

[env:esp32dev]

platform = https://github.com/tasmota/platform-espressif32/releases/download/v2.0.3rc1/platform-espressif32-2.0.3new.zip platform_packages = framework-arduinoespressif32 @ https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/826/framework-arduinoespressif32-v4.4_work-b98c8476f2.tar.gz

board = esp32dev framework = arduino

build_flags = -DCORE_DEBUG_LEVEL=3

; Set frequency to 240MHz. board_build.f_cpu = 80000000L

; Set partition scheme to https://github.com/espressif/arduino-esp32/tree/master/tools/partitions. ;FAIL board_build.partitions = esp32_partition_app2944k_spiffs2M.csv ;FAIL board_build.partitions = app3M_fat9M_16MB.csv ;PASS board_build.partitions = default.csv ;FAIL board_build.partitions = default_16MB.csv ;FAIL board_build.partitions = default_8MB.csv ;PASS board_build.partitions = default_ffat.csv ;FAIL board_build.partitions = ffat.csv ;PASS board_build.partitions = huge_app.csv ;FAIL board_build.partitions = large_spiffs_16MB.csv ;PASS board_build.partitions = min_spiffs.csv ;PASS board_build.partitions = minimal.csv ;PASS board_build.partitions = no_ota.csv ;PASS board_build.partitions = noota_3g.csv ;PASS board_build.partitions = noota_3gffat.csv ;PASS board_build.partitions = noota_ffat.csv ;PASS board_build.partitions = rainmaker.csv

; Set the flash size. board_build.flash_size = 8MB ; Set flash mode. board_build.flash_mode = dout ; Set flash frequency to 80MHz. board_build.f_flash = 40000000L

monitor_filters = esp32_exception_decoder monitor_speed = 115200

This is the very simple sketch that I am testing with:

#include "Arduino.h"

void setup() {
  log_e("setup");
}

void loop() {
  log_e("loop");
  vTaskDelay(1000);
}

[raphael-bmec-co]

reverting to:

platform = https://github.com/tasmota/platform-espressif32.git#v2.0.3rc1

They all work again. I think something is up with the Arduino Core build and/or the IDF?

[Jason2866]

Thats your problem

board_build.flash_size = 8MB

Board partition size and partition scheme have to fit together. You can NOT define board_build.flash_size without having the coresponding correct partition scheme Just do not use this setting! Define a custom partition scheme for your needs and use this!

All working with latest Arduino master :-)

[Jason2866]

Tried your test sketch with a M5 stack core2. No problem.

#include <Arduino.h>

#include <WiFi.h>
#include "esp_wifi.h"
#include "esp_heap_caps.h"

void setup() {
  heap_caps_malloc_extmem_enable(0);

  WiFi.begin("Jason_Home_WLAN", "xxxxx");
}

void loop() {
}

[env:m5stack-core2]
platform = https://github.com/tasmota/platform-espressif32/releases/download/v2.0.3rc1/platform-espressif32-2.0.3new.zip
platform_packages = framework-arduinoespressif32 @ https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/826/framework-arduinoespressif32-v4.4_work-b98c8476f2.tar.gz
board = m5stack-core2
board_build.partitions = esp32_partition_app2944k_spiffs10M.csv
framework = arduino
;lib_deps = ArduinoJson
build_flags =
   -DCORE_DEBUG_LEVEL=5
   -DBOARD_HAS_PSRAM=1
   -DCONFIG_ARDUINO_ISR_IRAM=1
   -O3 -ffast-math

M5 stack core2 has no PSRAM bug, so fixes not needed.

[.....öÍm.umïÕ¡Õ2-hal-cpu.c:214] setCpuFrequencyMhz(): PLL: 480 / 2 = 240 Mhz, APB: 80000000 Hz
[     4][I][esp32-hal-psram.c:96] psramInit(): PSRAM enabled
[    50][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 0 - WIFI_READY
[   139][V][WiFiGeneric.cpp:283] _arduino_event_cb(): STA Started
[   141][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 2 - STA_START
[   141][V][WiFiGeneric.cpp:96] set_esp_interface_ip(): Configuring Station static IP: 0.0.0.0, MASK: 0.0.0.0, GW: 0.0.0.0
[  3921][V][WiFiGeneric.cpp:295] _arduino_event_cb(): STA Connected: SSID: Jason_Home_WLAN, BSSID: 88:c3:97:b1:1d:56, Channel: 4, Auth: WPA2_PSK
[  3923][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 4 - STA_CONNECTED
[  5334][V][WiFiGeneric.cpp:305] _arduino_event_cb(): STA Got New IP:192.168.2.168
[  5335][D][WiFiGeneric.cpp:852] _eventCallback(): Arduino Event: 7 - STA_GOT_IP
[  5338][D][WiFiGeneric.cpp:914] _eventCallback(): STA IP: 192.168.2.168, MASK: 255.255.255.0, GW: 192.168.2.1

[Jason2866]

@SuGlider There is no issue

[raphael-bmec-co]

agreed that should match and I will remove it if not needed however even if it matches the boot loop still happens.

Taking a close look at the screenshot you send I noticed the sdk wasn't a 100% match. I took a closer look at the platform and platform_packages.

Changing the platform_packages did not help.

Changing the platform from platform-espressif32-2.0.3new.zip to platform-espressif32-2.0.3.zip (removing the new) made the issue go away.

Does this make sense to you? Is this a valid setup to test IDF changes?

Also I am using platform_packages release 826 but I see you are testing on 828. Do both of these have the IDF patch in them?

[Jason2866]

The difference is just minor. 826 is from yesterday and 828 from today. I tested both. Both work.

[raphael-bmec-co]

And the 2.0.3new versus 2.0.3?

[Jason2866]

It is only a minor update of how Platformio is catching the packages. Not related to the build process