search

apache / nuttx

Apache NuttX is a mature, real-time embedded operating system (RTOS)
https://nuttx.apache.org/
Apache License 2.0
2.8k stars 1.16k forks source link

[BUG] ESP32S3 spiflash_suspend_cache will hang at enable ESP32S3_APP_FORMAT_MCUBOOT #14368

Closed SmartIotMaker closed 1 week ago

SmartIotMaker commented 2 weeks ago

Description / Steps to reproduce the issue

CPU will hang when eanble ESP32S3_APP_FORMAT_MCUBOOT and ESP32S3_SPIFLASH，follow is console: ESP-ROM:esp32s3-20210327 Build:Mar 27 2021 rst:0x1 (POWERON),boot:0x2b (SPI_FAST_FLASH_BOOT) SPIWP:0xee mode:DIO, clock div:2 load:0x3fcd35a8,len:0x1528 d:0x403b0000,len:0x1dbc load:0x403ba000,len:0x2894 entry 0x403bc844 [esp32s3] [INF] Booting MCUboot build v1.10.0-131-gb206b99b [esp32s3] [INF] [boot] chip revision: v0.2 [esp32s3] [INF] [boot.esp32s3] Boot SPI Speed : 40MHz [esp32s3] [INF] [boot.esp32s3] SPI Mode : DIO [esp32s3] [INF] [boot.esp32s3] SPI Flash Size : 16MB [esp32s3] [INF] [boot] Enabling RNG early entropy source... [esp32s3] [INF] Primary image: magic=good, swap_type=0x1, copy_done=0x3, image_ok=0x1 [esp32s3] [INF] Scratch: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3 [esp32s3] [INF] Boot source: primary slot [esp32s3] [INF] Image index: 0, Swap type: none [esp32s3] [INF] Disabling RNG early entropy source... [esp32s3] [INF] br_image_off = 0x10000 [esp32s3] [INF] ih_hdr_size = 0x20 [esp32s3] [INF] Loading image 0 - slot 0 from flash, area id: 1
[esp32s3] [INF] DRAM segment: start=0x3265c, size=0x2208, vaddr=0x3fca8960 [esp32s3] [INF] IRAM segment: start=0x294c8, size=0x9194, vaddr=0x40374000 IROM segment aligned lma 0x00040000 vma 0x42020000 len 0x07d856 (514134) DROM segment aligned lma 0x00010000 vma 0x3c000000 len 0x0131ec (78316)

hang

then, remove spi_flash_disable_cache line in spiflash_suspend_cache function(esp32s3_spiflash.c) and spi_flash_restore_cache in spiflash_resume_cache function(esp32s3_spiflash.c) ， it is seem ok.

I check git log and find this change by commit 5680e9d5a4050f5efdeaf3965b685473d0b86234

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Ubuntu 24.04

NuttX Version

master

Issue Architecture

[Arch: xtensa]

Issue Area

[Area: Drivers]

Verification

tmedicci commented 1 week ago

Can you please provide more details on how to reproduce the issue? (compiler versions, exact commands and defconfig

SmartIotMaker commented 1 week ago

Run follow command:

  1. ./tools/configure.sh -S -l esp32s3-devkit:mcuboot_nsh
  2. make menuconfig to select Mount on bring-up" in "Board Selection" -> "Mount SPI Flash MTD on bring-up" -> "SMARTFS" , eanble Larg File Support in "File Systems" -> "Large File Support", enable BSD licenses in " License Setup"->"Use components that have BSD licenses", enable "
  3. make make bootloader to create mcuboot-esp32s3.bin file
  4. make -j to create nuttx.bin file
  5. esptool.py erase_flash
  6. make flash ESPTOOL_PORT=/dev/ttyUSB0 ESPTOOL_BINDIR=./
  7. minicom -b 115200 -D /dev/ttyUSB0 -o
  8. reset board by press reset button and then get follow log:
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x2b (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:2
load:0x3fcd35a8,len:0x1528
load:0x403b0000,len:0x1dbc
load:0x403ba000,len:0x2894
entry 0x403bc844
[esp32s3] [INF] *** Booting MCUboot build v1.10.0-131-gb206b99b ***
[esp32s3] [INF] [boot] chip revision: v0.2
[esp32s3] [INF] [boot.esp32s3] Boot SPI Speed : 40MHz
[esp32s3] [INF] [boot.esp32s3] SPI Mode       : DIO
[esp32s3] [INF] [boot.esp32s3] SPI Flash Size : 16MB
[esp32s3] [INF] [boot] Enabling RNG early entropy source...
[esp32s3] [INF] Primary image: magic=good, swap_type=0x1, copy_done=0x3, image_ok=0x1
[esp32s3] [INF] Scratch: magic=bad, swap_type=0x0, copy_done=0x2, image_ok=0x3
[esp32s3] [INF] Boot source: primary slot
[esp32s3] [INF] Image index: 0, Swap type: none
[esp32s3] [INF] Disabling RNG early entropy source...
[esp32s3] [INF] br_image_off = 0x10000
[esp32s3] [INF] ih_hdr_size = 0x20
[esp32s3] [INF] Loading image 0 - slot 0 from flash, area id: 1
[esp32s3] [INF] DRAM segment: start=0x15918, size=0xae0, vaddr=0x3fc8a180
[esp32s3] [INF] IRAM segment: start=0x12e58, size=0x2ac0, vaddr=0x40374000
[esp32s3] [INF] start=0x40374b54
IROM segment aligned lma 0x00020000 vma 0x42010000 len 0x01be6c (114284)
DROM segment aligned lma 0x00010000 vma 0x3c000000 len 0x002e18 (11800)

after that, no anymore info.

follow is defconfig context create bymake savedefconfig :

#
# This file is autogenerated: PLEASE DO NOT EDIT IT.
#
# You can use "make menuconfig" to make any modifications to the installed .config file.
# You can then do "make savedefconfig" to generate a new defconfig file that includes your
# modifications.
#
# CONFIG_ARCH_LEDS is not set
# CONFIG_NSH_ARGCAT is not set
# CONFIG_NSH_CMDOPT_HEXDUMP is not set
CONFIG_ALLOW_BSD_COMPONENTS=y
CONFIG_ARCH="xtensa"
CONFIG_ARCH_BOARD="esp32s3-devkit"
CONFIG_ARCH_BOARD_COMMON=y
CONFIG_ARCH_BOARD_ESP32S3_DEVKIT=y
CONFIG_ARCH_CHIP="esp32s3"
CONFIG_ARCH_CHIP_ESP32S3=y
CONFIG_ARCH_CHIP_ESP32S3WROOM1N4=y
CONFIG_ARCH_INTERRUPTSTACK=2048
CONFIG_ARCH_STACKDUMP=y
CONFIG_ARCH_XTENSA=y
CONFIG_BOARDCTL_RESET=y
CONFIG_BOARD_LOOPSPERMSEC=16717
CONFIG_BUILTIN=y
CONFIG_ESP32S3_APP_FORMAT_MCUBOOT=y
CONFIG_ESP32S3_SPIFLASH=y
CONFIG_ESP32S3_SPIFLASH_SMARTFS=y
CONFIG_ESP32S3_UART0=y
CONFIG_FS_PROCFS=y
CONFIG_FS_SPIFFS=y
CONFIG_HAVE_CXX=y
CONFIG_HAVE_CXXINITIALIZE=y
CONFIG_IDLETHREAD_STACKSIZE=3072
CONFIG_INIT_ENTRYPOINT="nsh_main"
CONFIG_INTELHEX_BINARY=y
CONFIG_NSH_ARCHINIT=y
CONFIG_NSH_BUILTIN_APPS=y
CONFIG_NSH_DISABLE_LOSMART=y
CONFIG_NSH_FILEIOSIZE=512
CONFIG_NSH_LINELEN=64
CONFIG_NSH_READLINE=y
CONFIG_PREALLOC_TIMERS=4
CONFIG_RAM_SIZE=114688
CONFIG_RAM_START=0x20000000
CONFIG_RR_INTERVAL=200
CONFIG_SCHED_WAITPID=y
CONFIG_START_DAY=6
CONFIG_START_MONTH=12
CONFIG_START_YEAR=2011
CONFIG_SYSLOG_BUFFER=y
CONFIG_SYSTEM_NSH=y
CONFIG_UART0_SERIAL_CONSOLE=y

when I trace step by step, I find cpu hang in spi_flash_disable_cache, follow is call stack:

init_storage_partition(esp32s3_board_spiflash.c) -> setup_spiffs(esp32s3_board_spiflash.c) ->  nx_mount(fs_mount.c) ->mops->bind()->spiffs_bind(spiffs_vfs.c) -> spiffs_objlu_scan(spiffs_core.c) -> spiffs_cache_read(spiffs_cache)
-> spiffs_mtd_read(spiffs_mtd.c) -> MTD_READ(fs->mtd, offset, len, dest) -> esp32s3_read(esp32s3_spiflash_mtd.c) -> spi_flash_read(esp32s3_spiflash.c) -> spiflash_start(esp32s3_spiflash.c) -> spiflash_suspend_cache(esp32s3_spiflash.c)
-> spi_flash_disable_cache(cache_hal.c)
SmartIotMaker commented 1 week ago

nuttx base on master and last commit is 204871513486e940fbb29a9a524de9b9e063d3e0 ：

commit 204871513486e940fbb29a9a524de9b9e063d3e0 (origin/master, origin/HEAD, master)
Author: Petro Karashchenko <petro.karashchenko@gmail.com>
Date:   Tue Oct 8 22:23:11 2024 +0200

    serial: remove 'TIOCSLINID'

    Signed-off-by: Petro Karashchenko <petro.karashchenko@gmail.com>

tool chain：

xtensa-esp32s3-elf-gcc -v
Using built-in specs.
COLLECT_GCC=/home/john/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20240906/xtensa-esp-elf/bin/xtensa-esp-elf-gcc
COLLECT_LTO_WRAPPER=/home/john/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20240906/xtensa-esp-elf/bin/../libexec/gcc/xtensa-esp-elf/14.2.0/lto-wrapper
Target: xtensa-esp-elf
Configured with: /builds/idf/crosstool-NG/.build/xtensa-esp-elf/src/gcc/configure --build=x86_64-build_pc-linux-gnu --host=x86_64-build_pc-linux-gnu --target=xtensa-esp-elf --prefix=/builds/idf/crosstool-NG/builds/xtensa-esp-elf --exec_prefix=/builds/idf/crosstool-NG/builds/xtensa-esp-elf --with-local-prefix=/builds/idf/crosstool-NG/builds/xtensa-esp-elf/xtensa-esp-elf --with-sysroot=/builds/idf/crosstool-NG/builds/xtensa-esp-elf/xtensa-esp-elf --with-native-system-header-dir=/include --with-newlib --enable-threads=no --disable-shared --with-pkgversion='crosstool-NG esp-14.2.0_20240906' --disable-__cxa_atexit --enable-cxx-flags=-ffunction-sections --disable-libgomp --disable-libmudflap --disable-libmpx --disable-libssp --disable-libquadmath --disable-libquadmath-support --disable-libstdcxx-verbose --with-gmp=/builds/idf/crosstool-NG/.build/xtensa-esp-elf/buildtools --with-mpfr=/builds/idf/crosstool-NG/.build/xtensa-esp-elf/buildtools --with-mpc=/builds/idf/crosstool-NG/.build/xtensa-esp-elf/buildtools --with-isl=/builds/idf/crosstool-NG/.build/xtensa-esp-elf/buildtools --enable-lto --enable-target-optspace --without-long-double-128 --disable-nls --enable-multiarch --enable-languages=c,c++ --disable-libstdcxx-verbose --enable-threads=posix --enable-gcov-custom-rtio --enable-libstdcxx-time=yes
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.2.0 (crosstool-NG esp-14.2.0_20240906)
tmedicci commented 1 week ago

Thanks for reporting: we were able to reproduce the error and we are testing a solution on our internal CI.