Very unstable system after migrating to Arduino-ESP32 3.0.01

[xwtk]

Board

ESP32-S3-WROOM-1-N8

Device Description

Custom hardware based on ESP32-S3-WROOM-1-N8 (240Mhz CPU set).

Hardware Configuration

GPIO 18, 19 allocated to HardwareSerial 1 Onewire BUS on GPIO 4 External accessory software power trigger on GPIO 5 (Output) Recovery trigger on GPIO 42 (Input with Pullup)

Version

v3.0.1

IDE Name

Arduino as ESP-IDF component

Operating System

Ubuntu 24.04 on ARM (RPi 400)

Flash frequency

N/A

PSRAM enabled

no

Upload speed

N/A

Description

Hey everyone.

I have recently migrated from ESP-IDF 4 with Arduino-ESP32 2.x.x to ESP-IDF 5 with Arduino-ESP32 3.0.1. I have read the migration guide and changed everything that needed to be done to run smoothly, however the system runs very unstable, details:

Certain standard ESP libraries such as Arduino main.cpp, Preferences, HardwareSerial are modified in my projects due to specific demands. In Arduino main.cpp, I remove loopTask and subsequent components of it in esp32-hal-rmt and esp32-hal-misc, while leaving CONFIG_AUTOSTART_ARDUINO set to true. I additionally have a 1MB encrypted NVS partition, which is used to its extent, since NVS stores a map in memory, I have to "mount" and "unmount" it whenever I need it, so instead of using Preferences.begin and .end, I modified and added a few functions which would keep the desired namespace stored in the private variable, and whenever an operation is called the function would init the nvs partition, enter namespace and do the action, afterwards deinit. I additionally made mount and unmount functions in case I would need to do bulk NVS operations with additional checks to prevent issues. I have an Asyncwebserver running with mDNS in background.

This scheme works perfectly with Arduino-ESP32 2.x.x (ESP-IDF 4), I have about 90-100K free heap available while everything is running except NVS, and when I need to do NVS operations, I initialize, do the action, and quit to leave the heap free. But with ESP-IDF 5 (Arduino-ESP 3.0.1), things have changed. Now if I mount the NVS once and unmount, whenever I try to do any change I get nvs_init failed with error "OTHER" and subsequently nvs_open failed with error "NOT_INITALIZED". While webserver and mDNS is running, a lot of the times ssl fails to allocate memory (all requests are done in HTTPS with some exceptions), note that this was NOT the case with Arduino-ESP32 2.x.x. As well as seems like heap memory got a hit, while previously I had 90-100K free, I now have 60-70K with systems running, on paper that shouldn't be a problem but seems like it is?

I mainly want to use Arduino-ESP32 3 (ESP-IDF 5) because I don't like using outdated versions, plus I may want to utilize Zigbee support later on. My suspicions fall on ESP RainMaker and ESP Skainet, which I don't utilize at all and don't know how to remove from the project.

Sketch

N/A

Debug Message

Provided in description

Other Steps to Reproduce

N/A

I have checked existing issues, online documentation and the Troubleshooting Guide

• [X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[lucasssvaz]

If you want to remove some specific components of the libs you can use Arduino as a component or recompile the libs using the Lib Builder

[xwtk]

If you want to remove some specific components of the libs you can use Arduino as a component or recompile the libs using the Lib Builder

Arduino is used as an ESP-IDF component. Arduino libraries are alright, what I suspect is RainMaker and Skainet which are baked into ESP-IDF, I am not sure how to remove them from the project.

[SuGlider]

@xwtk - a few questions:

1- What is the partition table used within Arduino 2.0.x and 3.0.x? NVS has its own partition and maybe there is no enough space after moving to IDF 5.1. 2- What are all the network services, resources and peripherals that your project uses? 3- Arduino as Component allow the user to change the sdkconfig of the project to better suit its needs. How differente is it from the one used in Arduino 2.0.x and 3.0.x?

[xwtk]

@xwtk - a few questions:

1- What is the partition table used within Arduino 2.0.x and 3.0.x? NVS has its own partition and maybe there is no enough space after moving to IDF 5.1. 2- What are all the network services, resources and peripherals that your project uses? 3- Arduino as Component allow the user to change the sdkconfig of the project to better suit its needs. How differente is it from the one used in Arduino 2.0.x and 3.0.x?

Same partition table, however it was flashed while on 2.0.x, afterwards upgrade to 3.0.x has been done only via OTA. No changes to the partition table or the project components except those added by ESP-IDF 5.1.4. Regarding network, WiFi, Asyncwebserver (HTTP), mDNS (broadcast).

[SuGlider]

Same partition table, however it was flashed while on 2.0.x, afterwards upgrade to 3.0.x has been done only via OTA.

As far as I remember, OTA only update the firmware. Therefore the partition information and NVS information are kept the same.

if I mount the NVS once and unmount, whenever I try to do any change I get nvs_init failed with error "OTHER" and subsequently nvs_open failed with error "NOT_INITALIZED".

Nothing was changed in Arduino regarding NVS. Given that the application doesn't use Preferences, this must be something related to IDF change from 4.x to 5.1.

I suggest checking IDF documentation.

While webserver and mDNS is running, a lot of the times ssl fails to allocate memory (all requests are done in HTTPS with some exceptions), note that this was NOT the case with Arduino-ESP32 2.x.x. As well as seems like heap memory got a hit, while previously I had 90-100K free, I now have 60-70K with systems running, on paper that shouldn't be a problem but seems like it is

You can disable PPP in IDF Menuconfig of your Arduino as IDF component project. That shall increase available HEAP.

[xwtk]

Same partition table, however it was flashed while on 2.0.x, afterwards upgrade to 3.0.x has been done only via OTA.

As far as I remember, OTA only update the firmware. Therefore the partition information and NVS information are kept the same.

if I mount the NVS once and unmount, whenever I try to do any change I get nvs_init failed with error "OTHER" and subsequently nvs_open failed with error "NOT_INITALIZED".

Nothing was changed in Arduino regarding NVS. Given that the application doesn't use Preferences, this must be something related to IDF change from 4.x to 5.1.

I suggest checking IDF documentation.

While webserver and mDNS is running, a lot of the times ssl fails to allocate memory (all requests are done in HTTPS with some exceptions), note that this was NOT the case with Arduino-ESP32 2.x.x. As well as seems like heap memory got a hit, while previously I had 90-100K free, I now have 60-70K with systems running, on paper that shouldn't be a problem but seems like it is

You can disable PPP in IDF Menuconfig of your Arduino as IDF component project. That shall increase available HEAP.

Correct, OTA doesn't change anything but the app firmware. NVS interaction is done via Preferences library which is modified to accommodate my needs, the modifications are explained in the initial post, I suspect the problem here comes from heap too, not sure though. Can you guide me where can I disable PPP in menuconfig? Thank you.

[davejel]

not sure if this is same / similar problem to mine with boards manager 3.0.1 i am using painlessmesh, with Arduino 2.3.2, after upgrading to boards manager (ESP32) to 3.0.1 none of my nodes would compile. downgraded to boards manager 2.0.17 and everything works again, contacted Rui Santos who says this is the best way to go for now but in his opinion painlessmesh is one of a few libs which are affected, does anyone know if painlessmesh will be updates to work with boards manager 3.0.1 please cheers Dave

[xwtk]

not sure if this is same / similar problem to mine with boards manager 3.0.1 i am using painlessmesh, with Arduino 2.3.2, after upgrading to boards manager (ESP32) to 3.0.1 none of my nodes would compile. downgraded to boards manager 2.0.17 and everything works again, contacted Rui Santos who says this is the best way to go for now but in his opinion painlessmesh is one of a few libs which are affected, does anyone know if painlessmesh will be updates to work with boards manager 3.0.1 please cheers Dave

Hey Dave, I am not using Arduino, rather native ESP-IDF with Arduino-ESP32 core as a component.

[davejel]

ok, thanks but it does show that 3.0.1 is causing problems on different platforms, commonality is saying something methinks

[xwtk]

Yes definitely. I also ran the same project with almost the same configuration (only changes done by IDF itself) on both ESP-IDF 4.4.6 (Arduino-ESP32 2.x.x) and ESP-IDF 5.1.4 (Arduino-ESP32 3.0.1), I have monitored and saw that 20-35kB of heap memory is lost! That is a hell of memory to be used while no new feature has been enabled. This is definitely caused by the garbage ESP-IDF Component Registry, the compiler adds literally as many components as it can, and nowhere is it written how to exclude them! Additionally PPP has always been disabled in the project, I double-checked to be sure. This is absolutely unacceptable, how did 3.0.0 and 3.0.1 get even marked as stable releases??

[SuGlider]

Can you guide me where can I disable PPP in menuconfig? Thank you.

@xwtk - did you manage to disable it in your project?

idf.py menuconfig Go in the Menu->Component config->LWIP seach for Enable PPP suport, press space bar on top of this option to uncheck it. press ESC, press S to save sdkconfig file (rename it or define a new folder for it), press ENTER, ESC, ESC and then build your project using the new sdkconfig modified file.

[SuGlider]

This is absolutely unacceptable, how did 3.0.0 and 3.0.1 get even marked as stable releases??

It is stable. As a new Version, it has breaking changes.

[xwtk]

Can you guide me where can I disable PPP in menuconfig? Thank you.

@xwtk - did you manage to disable it in your project?

idf.py menuconfig Go in the Menu->Component config->LWIP seach for Enable PPP suport, press space bar on top of this option to uncheck it. press ESC, press S to save sdkconfig file (rename it or define a new folder for it), press ENTER, ESC, ESC and then build your project using the new sdkconfig modified file.

As mentioned before, I checked and it was never enabled in the first place.

[xwtk]

This is absolutely unacceptable, how did 3.0.0 and 3.0.1 get even marked as stable releases??

It is stable. As a new Version, it has breaking changes.

Sure of course. But this isn't a breaking change, eating 20-35kB heap for nothing isn't normal. It is definitely a bunch of new components from the new ESP-IDF 5, but that should be something that can be removed in the first place.

[me-no-dev]

@xwtk everything that comes with the component is listed here: https://github.com/espressif/arduino-esp32/blob/master/idf_component.yml#L47-L94

You can comment out the components that you do not need. You also need to understand that not everything you see comes from us, IDF5 has many changes that increase ram/flash usage, but using it as component you could turn some off, like WPA3 and others.

[xwtk]

@xwtk everything that comes with the component is listed here: https://github.com/espressif/arduino-esp32/blob/master/idf_component.yml#L47-L94

You can comment out the components that you do not need. You also need to understand that not everything you see comes from us, IDF5 has many changes that increase ram/flash usage, but using it as component you could turn some off, like WPA3 and others.

Alright perfect, let me comment out unused modules and see if it changes anything.

[xwtk]

It increased heap a bit, but issues still remain.