Problem with SD_MMC in 1 bit mode (working fine with Arduino IDE)

[0x0fe]

So, i have an SD used with SDIO in 1-bit mode, it works perfectly with Arduino IDE, but the exact same sources fails in PIO. It initilizes without error, however when trying to write it immediately fails with error 265. The SOC is an ESP32D0WDR2 (V3) and i select esp32dev, same as on Arduino IDE. The firmware used for testing is strictly the same on both patforms. I added -DBOARD_HAS_1BIT_SDMMC to make sure it was in the correct mode, but it did not help at all.

below it the place where SD is initialized, but again there is no error at init. THe PIO is a brand new installation from yesterday, as up to date as can be. The 3 libraries are in the exact same version as on the Arduino IDE.

[env:esp32dev]
platform = https://github.com/platformio/platform-espressif32.git#v6.1.0
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.8
framework = arduino, espidf
board = esp32dev
board_build.f_flash = 80000000L
board_build.f_cpu = 240000000L
board_build.flash_mode = qio
board_upload.maximum_size = 3145728
board_build.partitions = huge_app.csv
board_upload.flash_size = 4MB
monitor_speed = 115200
monitor_filters = esp32_exception_decoder
build_flags = 
    -Wdouble-promotion
    -Wimplicit-fallthrough
    -DCONFIG_ARDUHAL_LOG_COLORS=1
    -DBOARD_HAS_PSRAM
    -DCORE_DEBUG_LEVEL=3
    -DCONFIG_MBEDTLS_PSK_MODES=1
    -DCONFIG_MBEDTLS_KEY_EXCHANGE_PSK=1
    -DCONFIG_FREERTOS_HZ=1000
    -DCONFIG_SPIRAM_SPEED_80M
    -DCONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP
    -DBOARD_HAS_1BIT_SDMMC
build_unflags = 
lib_deps = 
    esphome/ESP32-audioI2S@^2.0.7
    256dpi/MQTT@^2.5.1
    bblanchon/ArduinoJson@^6.21.2

E (13270) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x109
E (13271) diskio_sdmmc: sdmmc_read_blocks failed (265)
Error opening file
E (18304) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x109
E (18305) diskio_sdmmc: sdmmc_read_blocks failed (265)
Error opening file
E (22214) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x109
E (22215) diskio_sdmmc: sdmmc_read_blocks failed (265)
Error opening file
E (25368) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x109
E (25369) diskio_sdmmc: sdmmc_read_blocks failed (265)
Error opening file
E (29366) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x109
E (29367) diskio_sdmmc: sdmmc_read_blocks failed (265)
Error opening file
E (34305) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x109
E (34306) diskio_sdmmc: sdmmc_read_blocks failed (265)

here is the output from uart when the firmware is built by Arduino IDE, the SD_MMC is writing without problem, the speed shown here is the downlaod speed from aws.

13:03:20.494 -> CP01.mp3 [2060KB]
13:03:44.868 -> Done [24s 85KB/s]
13:03:48.732 -> CP02.mp3 [1234KB]
13:04:03.444 -> Done [14s 88KB/s]
13:04:07.405 -> CP03.mp3 [1381KB]
13:04:24.125 -> Done [16s 86KB/s]
13:04:27.429 -> CP04.mp3 [2561KB]
13:04:57.996 -> Done [30s 85KB/s]
13:05:01.766 -> CP05.mp3 [1715KB]

here is the piece of caode which actually writes to the SD_MMC, to be noted, the chunk size is very large (on purpose).

#define CHUNK_SIZE     65536

  while ( downloadRemaining > 0 && http.connected() ) {
    auto dstart = millis();
    auto data_size=stream->available();
    if (data_size > 0) {

      auto available_buffer_size = CHUNK_SIZE - (cur_buffer - buffer_);
      auto read_count = stream->read(cur_buffer, ((data_size > available_buffer_size) ? available_buffer_size : data_size));
      cur_buffer += read_count;
      downloadRemaining -= read_count;

      if (cur_buffer - buffer_ == CHUNK_SIZE) {
        auto wstart = millis();
        write(fileno(file), buffer_, CHUNK_SIZE);
        cur_buffer = buffer_;
        auto wend = millis();
        writeTime+=wend-wstart;
      }
    }
    auto dend = millis();
    downTime+=dend-dstart;     
    vTaskDelay(1);
    delayTime+=1;
  }

and here are the target options in arduino, all very standard:

[Jason2866]

Which core are you using with Arduino IDE? For Platformio you are targeting core 2.0.8 with settings not needed. Btw. You compare different things. ArduinoIDE is Arduino only. In Platformio you are compiling Arduino as component of IDF. You are comparing Apples vs. Pears

[0x0fe]

Hello, The arduino core is 2.0.8 as well.

For Platformio you are targeting core 2.0.8 with settings not needed

Which settings are not needed?

btw. You compare different things. ArduinoIDE is Arduino only. In Platformio you are compiling Arduino as component of IDF. You are comparing Apples vs. Pears

Uh, not sure what you are talking about, I did not compare anything. I know very well how platformio works, because i use it to allow easier changes in the core via menuconfig. As for your statement, it doesnt really change anything wether arduino layer is run as IDF component or wether arduino layer is run on top of a precompiled IDF core, in both case it is the same version of the core, same drivers and as far as i know sd_mmc driver is the same in both cases.

[Jason2866]

Replace this

[env:esp32dev]
platform = https://github.com/platformio/platform-espressif32.git#v6.1.0
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.8
framework = arduino, espidf
board = esp32dev

with

[env:esp32dev]
platform = espressif32 @ 6.3.1
framework = arduino
board = esp32dev

to use actual Arduino core 2.0.9. So it is the same as in ArduinoIDE

[0x0fe]

Oh ok, i will change this indeed.

By the way, i think i foudn the source of the problem (maybe) the error seems related to the block size when reading or writing, anything over 512K will lead to failure in PIO, as soon as i reduce to 512K it works.

#include <sdkconfig.h>
#include <Arduino.h>
#include "SD_MMC.h"

#define SD_D0 2
#define SD_CLK 14
#define SD_CMD 15

#define CHUNK_SIZE 512 // 65535

void testFileIO(fs::FS &fs, const char * path){

    File file = fs.open(path, FILE_READ);
    static uint8_t buf[CHUNK_SIZE];
    size_t len = 0;
    uint32_t start = millis();
    uint32_t end = start;
    uint32_t blocks = 1048576 / CHUNK_SIZE;

    if(file){
        len = file.size();
        Serial.printf("File length %u\n",len);
        size_t flen = len;
        start = millis();
        while(len){
            size_t toRead = len;
            if(toRead > CHUNK_SIZE){toRead=CHUNK_SIZE;}
            file.read(buf, toRead);
            len -= toRead;
        }
        end = millis() - start;
        Serial.printf("%u bytes read for %u ms\n", flen, end);
        Serial.printf("read %u KB/s\n", flen/end );
        file.close();
    } 
    else {
        Serial.println("Failed to open file for reading");
    }

    file = fs.open(path, FILE_WRITE);
    if(!file){
        Serial.println("Failed to open file for writing");
        return;
    }

    size_t i;
    start = millis();
    for(i=0; i<blocks; i++){
        file.write(buf, CHUNK_SIZE);
    }
    end = millis() - start;
    Serial.printf("%u bytes written for %u ms\n", blocks*CHUNK_SIZE, end);
    Serial.printf("write %u KB/s\n", (blocks*CHUNK_SIZE)/end );
    file.close();
}
void setup(void){

    Serial.begin(115200, SERIAL_8N1, -1, 1);
    disableCore1WDT();

    delay(5000);

    SD_MMC.setPins(SD_CLK, SD_CMD, SD_D0);
    while (!SD_MMC.begin("/sdcard", true, false, 40000)){
        printf("mount failed\n");
        delay(500);
    }

    uint8_t cardType = SD_MMC.cardType();
    if(cardType == CARD_NONE){
        Serial.println("No SD_MMC card attached");
        return;
    }

    Serial.print("SD_MMC Card Type: ");
    if(cardType == CARD_MMC){ Serial.println("MMC"); } 
    else if(cardType == CARD_SD){ Serial.println("SDSC"); } 
    else if( cardType == CARD_SDHC){Serial.println("SDHC"); } 
    else { Serial.println("UNKNOWN"); }

    uint64_t cardSize = SD_MMC.cardSize() / (1024 * 1024);
    Serial.printf("SD_MMC Card Size: %lluMB\n", cardSize);

    testFileIO(SD_MMC,"/test.txt");

    Serial.printf("Total space: %lluMB\n", SD_MMC.totalBytes() / (1024 * 1024));
    Serial.printf("Used space: %lluMB\n", SD_MMC.usedBytes() / (1024 * 1024));
}
void loop(void){

}

with CHUNK_SIZE defined to 1048, or 4096, 16384, 65635

SD_MMC Card Type: SDHC
SD_MMC Card Size: 32001MB
File length 1048576
Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.

Core  1 register dump:
PC      : 0x400d66d1  PS      : 0x00060030  A0      : 0x800d2b2b  A1      : 0x3ffb2140
A2      : 0x3ffc42a0  A3      : 0x00000000  A4      : 0x0000003f  A5      : 0x3ffb2140
A6      : 0x00000000  A7      : 0x00000008  A8      : 0x800d66a6  A9      : 0x3ffb2100
A10     : 0x0000001e  A11     : 0x00000040  A12     : 0x3f400194  A13     : 0x3ffb21c0
A14     : 0x3ffb21a0  A15     : 0x00000008  SAR     : 0x00000004  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x0000000c  LBEG    : 0x4008a159  LEND    : 0x4008a169  LCOUNT  : 0xffffffff

Backtrace: 0x400d66ce:0x3ffb2140 0x400d2b28:0x3ffb21e0 0x400d2e26:0x3ffb2230 0x400d6612:0x3ffb2260 0x40090295:0x3ffb2280

  #0  0x400d66ce:0x3ffb2140 in Print::printf(char const*, ...) at C:/Users/xxxx/.platformio/packages/framework-arduinoespressif32/cores/esp32/Print.cpp:50
  #1  0x400d2b28:0x3ffb21e0 in testFileIO(fs::FS&, char const*) at src/main_test.cpp:32
  #2  0x400d2e26:0x3ffb2230 in setup() at src/main_test.cpp:85
  #3  0x400d6612:0x3ffb2260 in loopTask(void*) at C:/Users/xxxx/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:42
  #4  0x40090295:0x3ffb2280 in vPortTaskWrapper at C:\Users\xxxx\.platformio\packages\framework espidf\components\freertos\port\xtensa/port.c:142

I when the CHUNK_SIZE was on larger values such as 16384 or 65635 i got backtraces related to queue errors too.

with CHUNK_SIZE defined to 512

SD_MMC Card Type: SDHC
SD_MMC Card Size: 32001MB
File length 1048576
1048576 bytes read for 352 ms
read 2978 KB/s
1048576 bytes written for 607 ms
write 1727 KB/s
Total space: 31968MB
Used space: 43MB

[0x0fe]

So after the changes in platformio.ini i can get my test to run with up to 32K buffer (CHUNK_SIZE 32768). Over this and it stops writing (49152 bytes), or crashes (>49152 bytes). However, when i try the firmware it stil fails to even init the card (i moved the card init to just before the mqtt init)

WiFi init
.
link up
IP: 192.168.1.25
fw rev 1.0.0
Stack Space: 6428
Used PSRAM: 7500
mac: xxxxxxxxxxxxxxxxxx
uuid: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ts: xxxxxxxxxxxxxxxx
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
E (10014) sdmmc_sd: sdmmc_init_sd_scr: send_scr (1) returned 0x109
E (10014) vfs_fat_sdmmc: sdmmc_card_init failed (0x109).
[ 10018][E][SD_MMC.cpp:138] begin(): Failed to initialize the card (0x109). Make sure SD card lines have pull-up resistors in place.
mount failed
E (10572) sdmmc_sd: sdmmc_init_sd_scr: send_scr (1) returned 0x109
E (10572) vfs_fat_sdmmc: sdmmc_card_init failed (0x109).
[ 10576][E][SD_MMC.cpp:138] begin(): Failed to initialize the card (0x109). Make sure SD card lines have pull-up resistors in place.
mount failed
E (11130) sdmmc_sd: sdmmc_init_sd_scr: send_scr (1) returned 0x109
E (11130) vfs_fat_sdmmc: sdmmc_card_init failed (0x109).
[ 11134][E][SD_MMC.cpp:138] begin(): Failed to initialize the card (0x109). Make sure SD card lines have pull-up resistors in place.
mount failed

So i will now integrate the MQTT and HTTPClient into the test and see at which point it fails. I suspect it may have to do with the wifi being active. The only other library which is used in the main firmware is esp32-audioi2s but i set the I2S pins manually (including MCK) so i know there is no conflict, i wil however add it to the bare test, in case the problem is related.

One noticable point is that i made changes into menuconfig related to wifi throughput and allocation to SPIRAM. I use the ESP32D0WDR which has internal 2MB PSRAM, in menuconfig i enabled the options to allow the wifi and bluetooth stack to allocate on the SPIRAM in priority (which it does, to a limited extent). Regarding the wifi throughut i modified lwip tcp parameters as below:

[Jason2866]

You can NOT change IDF config parameters when using Arduino only. All libs are precompiled with the values choosen in the Arduino Lib Builder. The used precompiled libs are the same for ArduinoIDE and Platformio.

[0x0fe]

i definitely can change IDF parameters when using arduino, because i did it several time, using arduino library builder in ubuntu under WSL,

Anyway that is not the point of this issue, this issue relates to platformio where i use menuconfig directly to change IDF options.

[Jason2866]

You OP was in ArduinoIDE it is working. There is 100% no way of change of menuconfig params! IF you compile your libs for ArduinoIDE yourself. You did nowhere mention this, you have to use the same framework build for Platformio.

Anyway that is not the point of this issue, this issue relates to platformio where i use menuconfig directly to change IDF options.

This fundamental info is missing in your OP.

[Jason2866]

I am not a beginner in building custom espressif32 frameworks and setups. The customized framework for project Tasmota is all my work.

[0x0fe]

I did not mention i changed IDF option in arduino core because i did not, i used a vanilla Arduino core to run these test, and they work fine. In the past i have modified the arduino core by recompiling the IDF libraries using arduino libraries builder. But it is not the case for this particular issue.

[Jason2866]

So it is not a platformio issue. The code base (precompiled libs) are identic since platformio uses the orig. release from Arduino espressif to build the pio framework. You can compare the precompiled libs. If it works in ArduinoIDE it will work with PlatformIO when all setting flags are same set.

[0x0fe]

I am not a beginner in building custom espressif32 frameworks and setups. The customized framework for project Tasmota is all my work.

Well, i never told so, not sure what this comment relates to. Still your statement

I open an issue for a specific problem with SD_MMC in PIO, and report facts. That's it.

[Jason2866]

There is no specific problem with SD_MMC in PIO. Arduino code base is the same for both. How should pio manage to introduce a code difference? Thats why i said the settings in pio choosen needs to be checked. That was my first post. Your posted pio setup is simply wrong.

[0x0fe]

So it is not a platformio issue. The code base (precompiled libs) are identic since platformio uses the orig. release from Arduino espressif to build the pio framework. You can compare the precompiled libs.
If it works in ArduinoIDE it will work with PlatformIO when all setting flags are same set.

So, instead of checking the end result, you make bold and definitive assumptions, leading you to tell : the problem doesnt exist. Alright, why do you think i open a ticket then? As if there was not mutlipel issues in the past of things not working in PIO and working fine in Arduino...

[0x0fe]

There is no specific problem with SD_MMC in PIO. Arduino code base is the same for both. How should pio manage to introduce a code difference?
Thats why i said the settings in pio choosen needs to be checked. That was my first post. Your posted pio setup is simply wrong.

Yeah, ok, that you tell, however the SD_MMC still fails, as you can see on the log. PIO settings have been modified, this had no impact on the issue. I will continue testing, also when i tested the first time all menuconfig options were to their default values. It cannot be assumed there is no difference between PIO and Arduino, even when runnning the same code with same libraries, otherwise it would run exactly the same and it doesnt.

[Jason2866]

As if there was not mutlipel issues in the past of things not working in PIO and working fine in Arduino...

Tell me one issue in platformio which is Arduino code base related (as it is your example).

You wont find one. Yes there where bugs in build system, flashing and other thing which needs to be correct to get it going.

It cannot be assumed there is no difference between PIO and Arduino, even when runnning the same code with same libraries, otherwise it would run exactly the same and it doesnt.

Your setup (you posted) is not the same!! Told already

[0x0fe]

By the way, after the PIO update following the platformio.ini modification the menuconfig quick access button has vanished, trying to call pio run -t menuconfig on the terminal also fails.

[Jason2866]

In Arduino only setup there is no menuconfig. Makes no sense to have since nothing can be configured.

[0x0fe]

i dont know who says "nothing can be configured" in the core, with arduino, changing option via menuconfig, recompiling the core and replacing the libraries inside arduino is the standard way to proceed, and streamlining this process is the very reason i used PIO here.

So i understand now that my platformio.ini setting was not incorrect, it was the correct way to have menuconfig and i'll revert to :

framework = arduino, espidf

I will also revert sdkconfig to the default and run the sd_mmc test again.

[0x0fe]

so, this

platform = espressif32 @ 6.3.1
framework = arduino, espidf
board = esp32dev

Leads to immediate crash at early stage of boot.

I suspect the IDF in the project and the expected IDF version by arduino32 differ, or something like that, so i will revert to what worked, it was set to arduino-esp32 2.0.8 already anyway (current version on arduino platform where the firmware works.).

[env:esp32dev]
platform = https://github.com/platformio/platform-espressif32.git#v6.1.0
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.8
framework = arduino, espidf

[0x0fe]

back to the initial platformio.ini configuration (6.1.0 and 2.0.8), default sdkconfig file,, sd_mmc still fails.

[valeros]

Hi 0x0fe, you're comparing two different setups when reproducing the issue as @Jason2866 already correctly pointed out. Please try the following platformio.ini which corresponds to the latest ESP32 core in Arduino IDE:

[env:esp32dev]
platform = espressif32 @ 6.3.1
framework = arduino
board = esp32dev
...

Note: It's expected that you won't be able to run the menuconfig target as it's only available in mixed IDF/Arduino projects which cannot be directly compared to projects running successfully in Arduino IDE.

[0x0fe]

@valeros well, it has been done yesterday, and all the issues reproduced exactly the same on 6.3.1 For me using framework = arduino, espidf is necessary because i need menuconfig, it is the whole point of using PIO, many options have t be adjusted later for wifi throughput and SPIRAM allocations for the BT/wifi stacks.

platform = espressif32 @ 6.3.1

jason2866 told it correspond with core 2.0.9, but in arduino i am on 2.0.8, and the setting i have is also 2.0.8, so i am not sure who is correct.

platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.8

Anyway, there is definitely some issues with sd_mmc, I2S, DMA and PSRAM, which do not occur at all in Arduino with the exact same test firmware, core, library. I suspect the sd_mmc, I2S, DMA issues are somewhat related to PSRAM too, which shouldn't be since these are on IRAM.

[valeros]

jason2866 told it correspond with core 2.0.9, but in arduino i am on 2.0.8, and the setting i have is also 2.0.8, so i am not sure who is correct.

My bad, platform version 6.2.0 uses core v2.0.8, please try the following:

platform = espressif32 @ 6.2.0

For me using framework = arduino, espidf is necessary because i need menuconfig

Anyway, there is definitely some issues with sd_mmc, I2S, DMA and PSRAM, which do not occur at all in Arduino with the exact same test firmware, core, library. I suspect the sd_mmc, I2S, DMA issues are somewhat related to PSRAM too, which shouldn't be since these are on IRAM.

In this case we cannot directly compare the behavior with Arduino IDE as there are simply two different build processes. I'd recommend preparing a minimal project example that doesn't need extra menuconfig settings, but clearly shows the issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Please provide more details or it will be closed if no further activity occurs. Thank you for your contributions.