zephyrproject-rtos / zephyr

Primary Git Repository for the Zephyr Project. Zephyr is a new generation, scalable, optimized, secure RTOS for multiple hardware architectures.
https://docs.zephyrproject.org
Apache License 2.0
10.87k stars 6.62k forks source link

Could not get SD card mutex #63136

Closed hongquan-prog closed 1 year ago

hongquan-prog commented 1 year ago

Describe the bug I have turned on the option to use files to save logs, this will cause the log thread to write logs to the SD card when the system is idle, when I read or write to the file in other threads, the fs_read function returns -5, with a message that it could not acquire the mutex lock(relative path: subsys/sd/sd_ops.c:card_read_blocks), here's the result I get after some simple debugging:

20230927-142301

  1. first of all, I printed the TCB pointer of the lock and the TCB pointer of the current access function.
  2. then I printed out the TCB pointers of all threads in the system to find the thread that holds the lock, from the figure you can see that the logging thread holds the lock, resulting in the system read and write file failure.
  3. due to the mutual exclusion lock is no wait, as long as there is read and write disk operation is interrupted, will cause other thread read and write file failure.
  4. the higher the frequency of reading or writing files, the greater the probability of this error

To Reproduce Steps to reproduce the behavior:

  1. Prepare an SD card and a imxrt1061 board with an SD card slot.
  2. Turn on options related to file logging
    
    CONFIG_DISK_DRIVER_SDMMC=y
    CONFIG_SD_INIT_PRIORITY=90
    CONFIG_SDMMC_VOLUME_NAME="SD"
    CONFIG_SDMMC_SUBSYS=y
    CONFIG_SDMMC_LOG_LEVEL_INF=y
    CONFIG_SDMMC_LOG_LEVEL=3

SDHC

CONFIG_SDHC=y CONFIG_IMX_USDHC=y CONFIG_IMX_USDHC_DAT3_PWR_TOGGLE=y CONFIG_IMX_USDHC_DMA_SUPPORT=y CONFIG_SDHC_BUFFER_ALIGNMENT=4 CONFIG_IMX_USDHC_DMA_BUFFER_SIZE=128 CONFIG_SDHC_INIT_PRIORITY=85 CONFIG_SDHC_SUPPORTS_UHS=y CONFIG_SDHC_SUPPORTS_NATIVE_MODE=y CONFIG_SDHC_LOG_LEVEL_INF=y CONFIG_SDHC_LOG_LEVEL=3

SDMMC STACK

CONFIG_SDMMC_STACK=y CONFIG_SD_STACK=y CONFIG_SD_LOG_LEVEL_INF=y CONFIG_SD_LOG_LEVEL=3 CONFIG_SD_INIT_TIMEOUT=1500 CONFIG_SD_RETRY_COUNT=10 CONFIG_SD_OCR_RETRY_COUNT=1000 CONFIG_SD_CMD_TIMEOUT=200 CONFIG_SD_DATA_TIMEOUT=10000 CONFIG_SD_BUFFER_SIZE=512 CONFIG_SD_DATA_RETRIES=3 CONFIG_SD_UHS_PROTOCOL=y CONFIG_MMC_RCA=2

FS LOGGING

CONFIG_LOG=y CONFIG_LOG_BUFFER_SIZE=4096 CONFIG_DISK_ACCESS=y CONFIG_FAT_FILESYSTEM_ELM=y CONFIG_LOG_BACKEND_FS=y CONFIG_LOG_BACKEND_FS_AUTOSTART=y CONFIG_LOG_BACKEND_FS_DIR="/SD:" CONFIG_LOG_BACKEND_FS_FILES_LIMIT=1000 CONFIG_LOG_BACKEND_FS_FILE_SIZE=20480 CONFIG_LOG_BACKEND_FS_FILE_PREFIX="log"


3. Write a thread to output logs, the more often the logs are output the better
4. Write a thread that reads the file, the more frequently it is read the easier it is to reproduce the problem
5. See error (Could not get SD card mutex)

**Expected behavior**
No errors are returned during file reading

**Impact**
Unable to read and write files in a multi-threaded environment, the system can get file read and write errors at any time

**Environment (please complete the following information):**
 - Linux
 - Zephyr SDK 0.15.2
 - SHA: 909f9650321a42b40aab57763bc3ff8dd32ce079
github-actions[bot] commented 1 year ago

Hi @hongquan-prog! We appreciate you submitting your first issue for our open-source project. 🌟

Even though I'm a bot, I can assure you that the whole community is genuinely grateful for your time and effort. 🤖💙

nordicjm commented 1 year ago

Did you enable CONFIG_FS_FATFS_REENTRANT ?

hongquan-prog commented 1 year ago

The company develops based on version 3.3, the option does not exist in zephyr 3.3 yet, is there a patch for it?

nordicjm commented 1 year ago

No, you'd either have to move to 3.4 (if it has it) or main, or find the commit(s) and cherry-pick them

hongquan-prog commented 1 year ago

No, you'd either have to move to 3.4 (if it has it) or main, or find the commit(s) and cherry-pick them OK,thanks for your help