espressif / arduino-esp32

Arduino core for the ESP32
GNU Lesser General Public License v2.1
13.67k stars 7.42k forks source link

Persistent Boot Failure on ESP32-S3 #9874

Closed Gspohu closed 4 months ago

Gspohu commented 5 months ago

Board

ESP32-S3-WROOM-1U-N4R2

Device Description

Plain module

Hardware Configuration

Irrelevant

Version

v3.0.1

IDE Name

PlatformIO

Operating System

Linux Ubuntu 22.04

Flash frequency

80

PSRAM enabled

yes

Upload speed

115200

Description

Hello,

I am experiencing a consistent boot failure issue with my ESP32-S3, regardless of the complexity or simplicity of the firmware I attempt to upload. This issue persists across both my pretty complex custom firmware and basic example code, such as the WiFi connection example. The behavior suggests a fundamental problem related to the boot process.

Issue Description

The device fails to boot properly. The error message that is consistently output is as follows:

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40377989
...
E (681) mmap: esp_mmu_unmap(595): munmap target pointer is outside external memory regions
assert failed: spi_flash_munmap flash_mmap.c:241 (0 && "invalid handle, or handle already unmapped")
Backtrace: 0x40377d1e:0x3fcf4610 ...
ELF file SHA256: fd4bb4dfa7adf1ca

My point of view is that it's linked to the model chip I have.

I'm currently stuck on this model of chip, but it's not the first time I've had to unravel memory-related boot problems with this model of chip ESP32-S3-WROOM-1U-N4R2.

Additional Context

This seems to be a regression or a newly introduced bug associated with the latest updates. Any guidance or suggestions on how to mitigate this issue would be greatly appreciated. Thank you for your attention to this matter.

Sketch

None

Debug Message

rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40377989
SPIWP:0xee
mode
, clock div:1
load:0x3fce3818,len:0x508
load:0x403c9700,len:0x4
load:0x403c9704,len:0xad0
load:0x403cc700,len:0x29e4
entry 0x403c9880
E (681) mmap: esp_mmu_unmap(595): munmap target pointer is outside external memory regions

assert failed: spi_flash_munmap flash_mmap.c:241 (0 && "invalid handle, or handle already unmapped")

Backtrace: 0x40377d1e:0x3fcf4610 0x4037d915:0x3fcf4630 0x403838a5:0x3fcf4650 0x4201d19b:0x3fcf4780 0x4201cbd1:0x3fcf47a0 0x4201ccd5:0x3fcf47c0 0x4201d06f:0x3fcf47f0 0x42014784:0x3fcf4850 0x420167a2:0x3fcf4890 0x420c4ac9:0x3fcf48c0 0x4038064a:0x3fcf4900

ELF file SHA256: fd4bb4dfa7adf1ca

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

me-no-dev commented 5 months ago

@Gspohu I don't have this specific flavor, but you are the first to report such issue. Does it work fine if you go back to 2.0.17?

@igrr any ideas? Maybe should be moved to ESP-IDF?

Gspohu commented 5 months ago

Yes, I looked carefully before posting the issue and it seems that I'm the first with this problem.

I can't go back to version 2, I've already migrated my firmware to version 3 and it's not possible to build it.

As I'm an idiot, when I updated I didn't look to see which commit I was on...

I'm currently testing them one by one to see when the bug appeared. If I find it, I'll post the info here.

If it can help here my plaformio.ini :

[env:esp32-s3-N4]
framework = arduino
board = adafruit_feather_esp32s3
monitor_speed = 9600
platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
  framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git
  framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#idf-release/v5.1
lib_deps =
  bblanchon/ArduinoJson@^6.19.4
  WiFi

build_flags = 
    -DCONFIG_LWIP_IPV4_NAPT=1
    -DIP_NAPT=CONFIG_LWIP_IPV4_NAPT
    -DCONFIG_LWIP_IP_FORWARD=1
    -DCONFIG_LWIP_L2_TO_L3_COPY=1
    -DCONFIG_IDF_TARGET_ESP32S3=1
    -DARDUINO_USB_MODE=1
    -DARDUINO_USB_CDC_ON_BOOT=1
    -DCONFIG_LWIP_IPV6=1
    -DCONFIG_LWIP_IPV6_DHCP6=1
    -DCONFIG_LWIP_IPV6_AUTOCONFIG=1
Jason2866 commented 5 months ago

remove the not working settings in build_flags. This are compile time settings only working with IDF. The are useless with Arduino. Pin the used Arduino libs with commit SHA

[env:esp32-s3-N4]
framework = arduino
board = adafruit_feather_esp32s3
monitor_speed = 9600
platform = https://github.com/platformio/platform-espressif32.git#40cf417aa2e2d32b32c5e9540c458e8869c5ec58
platform_packages =
  framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#cbf1e94dd5f9ac8bdabaae9603611a3b81d8fbe4
  framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#a51b0d7fd9bffef217f98d1bc0bedca327bc83ea
lib_deps =
  bblanchon/ArduinoJson@^6.19.4
build_flags = 
    -DARDUINO_USB_MODE=1
    -DARDUINO_USB_CDC_ON_BOOT=1

I personally wouldn't use the Adafruit boards.json at all since it uses the imho flash wasting tinyuf2

Gspohu commented 5 months ago

OK I'm going to test the build flag, but it's not my choice to use Adafruit board json, for this specific chip I don't know what to use as a board file. If you have one to recommend to me, I'm open !

Edit : I've removed the non-arduino build flag and that doesn't change the problem

Jason2866 commented 5 months ago

copy this one in folder boards in root of your project and give it a name like S3-WROOM-1U-N4R2.json to use in Platformio: board = S3-WROOM-1U-N4R2

{
  "build": {
    "arduino": {
      "memory_type": "qio_qspi"
    },
    "core": "esp32",
    "extra_flags": [
      "-DBOARD_HAS_PSRAM",
      "-DARDUINO_RUNNING_CORE=1",
      "-DARDUINO_EVENT_RUNNING_CORE=1"
    ],
    "f_cpu": "240000000L",
    "f_flash": "80000000L",
    "flash_mode": "qio",
    "mcu": "esp32s3",
    "variant": "esp32s3"
  },
  "connectivity": [
    "wifi",
    "bluetooth"
  ],
  "debug": {
    "openocd_target": "esp32s3.cfg"
  },
  "frameworks": [
    "arduino",
    "espidf"
  ],
  "name": "WROOM-1-N4R2",
  "upload": {
    "flash_size": "4MB",
    "maximum_ram_size": 327680,
    "maximum_size": 4194304,
    "require_upload_port": true,
    "speed": 460800
  },
  "url": "https://www.espressif.com",
  "vendor": "espressif"
}
Jason2866 commented 5 months ago

If it still bootloops one of your used libraries is to blame. Probably the breaking change PR https://github.com/espressif/arduino-esp32/pull/9453 introduces problems in many extern libraries

TD-er commented 5 months ago

Can you disable anything related to PSRAM? Yesterday a user of my software also reported some issues (unclear yet what actually is the issue), and he had also one of the newer S3 chips with 4M flash and 2M PSRAM. (his board uses ESP32-S3FH4R2 ) And since the S3 does have some issues with adressing flash/PSRAM, maybe disabling PSRAM is worth a try to narrow down the issue.

me-no-dev commented 5 months ago

H4R2 is High Temp flash and psram, N4R2 is Normal Temp (85C). @TD-er can you get more info from your user?

TD-er commented 5 months ago

@me-no-dev I will ask him to do some more tests to figure out what actually went wrong.

As far as I know is that he reported he is unable to flash this board using the web flasher. But I figured that if he also might have some bootlooping board after an initial flash, that it would easily look like "unable to flash" It is this board: image

Gspohu commented 5 months ago

I've tested it with @Jason2866 new board file (thanks for that), but the bug is exactly the same.

BUT I disabled the PSRAM by removing the flag in the board file and the problem disappeared. So it's the PSRAM management that's at fault here.

It's a bit off the subject for this issue (if I find the time I'll do an issue on it), but on master the WiFi isn't working at all, and it's impossible to enable WPS. I think it's because there is no more ram

TD-er commented 5 months ago

How was the PSRAM configured? I think it was configured to use OPI wiring.

Can you compare your board JSON with what I'm using for 4M S3 boards:

{
  "build": {
    "arduino":{
      "ldscript": "esp32s3_out.ld",
      "memory_type": "qio_qspi"
    },
    "core": "esp32",
    "extra_flags": "-DARDUINO_TASMOTA -DBOARD_HAS_PSRAM -DARDUINO_USB_MODE=1 -DUSE_USB_CDC_CONSOLE -DESP32_4M -DESP32S3 -DARDUINO_USB_CDC_ON_BOOT=1",
    "f_cpu": "240000000L",
    "f_flash": "80000000L",
    "flash_mode": "qio",
    "hwids": [
      [
        "0x303A",
        "0x1001"
      ]
    ],
    "mcu": "esp32s3",
    "variant": "esp32s3",
    "partitions": "boards/partitions/esp32_partition_app1810k_spiffs316k.csv"
  },
  "connectivity": [
    "wifi",
    "bluetooth",
    "ethernet"
  ],
  "debug": {
    "default_tool": "esp-builtin",
    "onboard_tools": [
      "esp-builtin"
    ],
    "openocd_target": "esp32s3.cfg"
  },
  "frameworks": [
    "espidf",
    "arduino"
  ],
  "name": "Espressif Generic ESP32-S3 USB CDC >= 4M Flash QSPI PSRAM, ESPEasy 1810k Code/OTA 316k FS",
  "upload": {
    "flash_size": "4MB",
    "maximum_ram_size": 327680,
    "maximum_size": 4194304,
    "require_upload_port": true,
    "before_reset": "usb_reset",
    "speed": 460800
  },
  "url": "https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/hw-reference/esp32s3/",
  "vendor": "Espressif"
}

You may have some options different, like partitions and maybe some specific flags.

Gspohu commented 5 months ago

You can see mine here : https://github.com/espressif/arduino-esp32/issues/9874#issuecomment-2173681779 I just remove "-DBOARD_HAS_PSRAM",

TD-er commented 5 months ago

Then I honestly have no idea why it would not work with PSRAM enabled.

Gspohu commented 4 months ago

I took the time to make a more methodical study of the problem, to understand when it first appeared.

So I went back to the version where I was before the problem appeared. Here's a table summarising my research/tests: Date Commit Build ? Start ? WiFi ? WPS ? Note Note
29/04/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#7e7c01aadf81af799ab296574827d16ddfe8621f
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#b0850286f4be64cb08eeedcf1594cbd5b59ea7f1
Yes Yes Yes Yes
02/05/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#5a023568a640f7b55f40381a28389d05df3af9c5
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#9250b6970c27d2863935fad42fd8d491b4731ff1
Yes Yes Yes Yes
07/05/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#789b1a84258e1b50f40d651be8a3f39226c98753
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#831d0d92c7c998493dbc379798b8efe4c617b300
Yes Yes Yes Yes
08/05/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#e883a2ccfac32b64b50926a3c8a498745a9546de
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#07d6415e1df493c23a3091761be59ece49527313
Yes Yes Yes Yes
13/05/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#0c4b35e07ea696a94de859151c00acbef12eda7f
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#951ade74d7886e1ce931ea46614c4ac47ae3a6c0
Yes Yes Yes Yes
22/05/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#d164df89de350aa945ccb0cb4261f56306bfdb3c
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#8206d4ee579b83a955ba22dfa45588c836522c0b
Yes Yes Yes Yes
29/05/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#79b835059f6a2f39b74dde92144909a52617ee5f
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#db132bfa666a5b1705ec810d9addbb8aaede2b33
Yes Yes Yes Yes
30/05/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#c16a325439fe5a8a3160701d9601634155bb1443
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#80ef50f35ae7f6736ad639c236c846cac8a9750e
Yes Yes Yes Yes
04/06/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#640523827be8db2ed201623f107569a6dd1b2a13
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#a6dac936bfcfa4c11468e003f550b41c76efeae6
Yes Yes Yes Yes
05/06/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#cc50d90ce4ff7414015dfc597eb083d427b86faf
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#a6dac936bfcfa4c11468e003f550b41c76efeae6
Yes Yes Yes Yes Release 3.0.1
06/06/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#5d9cd9ac58471624c07295112095cc3864b9cbaf
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#a6dac936bfcfa4c11468e003f550b41c76efeae6
Yes Yes ? ? esp32-arduino-libs of 04/06/2024 Impossible to monitor, USB or UART. But the LED start and is green, so we can think that the WiFi is connected
07/06/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#e3fedc5e8c47427d2fc8c4b45c60dd083243d786
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#a6dac936bfcfa4c11468e003f550b41c76efeae6
Yes Yes ? ? esp32-arduino-libs of 04/06/2024 Impossible to monitor in USB but in UART I can see somme debug line about the WiFi but that's all, nothing more. But the LED start and is green, so we can think that the WiFi is connected
10/06/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#575a415719d875f88460ec1461408597744f7053
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#a6dac936bfcfa4c11468e003f550b41c76efeae6
Yes Yes ? ? esp32-arduino-libs of 04/06/2024 Impossible to monitor in USB but in UART I can see somme debug line about the WiFi but that's all, nothing more. But the LED start and is green, so we can think that the WiFi is connected
12/06/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#e382746b95be9f19a74a10408ae3e250e36e1930
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#a6dac936bfcfa4c11468e003f550b41c76efeae6
Yes Yes ? ? esp32-arduino-libs of 04/06/2024 Impossible to monitor in USB but in UART I can see somme debug line about the WiFi but that's all, nothing more. But the LED start and is green, so we can think that the WiFi is connected
13/06/2024 platform = https://github.com/platformio/platform-espressif32.git
platform_packages =
framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#cbf1e94dd5f9ac8bdabaae9603611a3b81d8fbe4
framework-arduinoespressif32-libs@https://github.com/espressif/esp32-arduino-libs.git#9f889b1dadfb54c37f017b21f70c35828941a3fd
Yes No, crash as in this issue No No

The error message I talk about is :

[ 5584][E][NetworkManager.cpp:130] hostByName(): DNS Failed for 'xxx.eu' with error '-54'
[ 5594][E][ssl_client.cpp:116] start_ssl_client(): connect on fd 48, errno: 118, "Host is unreachable"
[ 5604][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[ 5613][E][NetworkClient.cpp:315] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[ 5621][E][NetworkClient.cpp:315] setSocketOption(): fail on 0, errno: 9, "Bad file number"
E (21109) wifi:sta is connecting, return error

My board file is exactly this one : https://github.com/espressif/arduino-esp32/issues/9874#issuecomment-2173681779

In conclusion, the start problems linked to PSRAM really start between commit e382746 and commit cbf1e94 dated 13/06/2024.

If this speaks to anyone, I'm listening, because I don't know this code base enough to draw anything from my study. Lastly, is it possible that this is linked to precompiled libs?

me-no-dev commented 4 months ago

is it possible that this is linked to precompiled libs?

It is, because we did not change anything that can influence the result. You say you can not debug, how come? What do you get in Serial?

Gspohu commented 4 months ago

Thanks for the feedback, @me-no-dev.

You mentioned that the issues might be linked to precompiled libraries, as there have been no changes that could directly affect this outcome. However, I'm facing a lack of serial output post-crash, which complicates pinpointing the exact cause.

Here the only message I get from the serial :

rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40377989
SPIWP:0xee
mode
, clock div:1
load:0x3fce3818,len:0x508
load:0x403c9700,len:0x4
load:0x403c9704,len:0xad0
load:0x403cc700,len:0x29e4
entry 0x403c9880
E (681) mmap: esp_mmu_unmap(595): munmap target pointer is outside external memory regions

assert failed: spi_flash_munmap flash_mmap.c:241 (0 && "invalid handle, or handle already unmapped")

Backtrace: 0x40377d1e:0x3fcf4610 0x4037d915:0x3fcf4630 0x403838a5:0x3fcf4650 0x4201d19b:0x3fcf4780 0x4201cbd1:0x3fcf47a0 0x4201ccd5:0x3fcf47c0 0x4201d06f:0x3fcf47f0 0x42014784:0x3fcf4850 0x420167a2:0x3fcf4890 0x420c4ac9:0x3fcf48c0 0x4038064a:0x3fcf4900

ELF file SHA256: fd4bb4dfa7adf1ca

And it's restart loop like this.

Here’s a quick rundown of what I’m observing:

Could you clarify if there have been any recent changes in the ESP-IDF that might affect memory or PSRAM handling specifically? Or where I can search for that ? Maybe I need to post my issue somewhere else ?

Also, are there recommended tools or methods for diagnosing this issue more effectively in the absence of clear serial debug outputs?

me-no-dev commented 4 months ago

Could you clarify if there have been any recent changes in the ESP-IDF that might affect memory or PSRAM handling specifically?

I can not. We can not follow the changes in ESP-IDF. Big project, many people and changes.

Maybe I need to post my issue somewhere else ?

ESP-IDF's issue tracker is a good place to post such issue with all details that you have figured out thus far. 2.0.17 is on IDF v4.4.7, 3.0.1 is on IDF v5.1.4

Gspohu commented 4 months ago

Ok, I will try to post my issue over there.

https://github.com/espressif/esp-idf/issues/14035