Partition table support on SD card (IDFGH-12063)

[huming2207]

Is your feature request related to a problem?

Hi all,

Currently for reading and writing SD card with filesystems, we either can use FAT filesystem which is up to 32GB, or use LittleFS which is up to 2GB. But modern SD cards are often bigger than that. In order to make the full use of the whole SD card, I'd suggest adding support for partition table.

cc @adokitkat

Describe the solution you'd like.

• Using esp_paritition is possible, but it needs quite a bit hacks. The size and address fields in esp_partition are all uint32_t which is limited to 4GB. I guess Espressif initially only consider using this library within their SPI flash scope, not for something bigger like SD cards.
• Porting MBR partition support is also doable, and I think it's probably the most useful option, so then SD card is accessible on a normal computer with a SD card reader.
• It would be great to support GPT (GUID partition table), but AFAIK it's harder and it's heavier for RAM consumption when parsing

Describe alternatives you've considered.

N/A

Additional context.

Here's an existing discussion: https://github.com/joltwallet/esp_littlefs/issues/173

[adokitkat]

Hi. Just a note:

If I understand it correctly, the 32GB limit of a FAT32 filesystem is only a fake limit pushed by Windows. FAT32 should actually support up to 16TB (and I read most operating systems permit up to 2TB). You can use cmd commands or a custom software to format 32GB+ SD cards on a PC and you should be able to use the format function in ESP-IDF as well (although I would disable the Task WDT in idf.py menuconfig so it won't trigger mid-formatting and reset the ESP - on ESP it will take some time with a such large capacity).

cc @pacucha42

[huming2207]

If I understand it correctly, the 32GB limit of a FAT32 filesystem is only a fake limit pushed by Windows. FAT32 should actually support up to 16TB (and I read most operating systems permit up to 2TB).

Ah yes... I forgot this matter! Indeed it's 16TB if sector size is 4KB.

But meanwhile FAT32 is a bad choice for embedded stuff. The partition may be corrupted if there's a power loss during write operations. LittleFS is a better choice but it's limited to 2GB. Thus I think partitioning is a better option.

[adokitkat]

Yes it kind of is... We want to address it and are developing a journaling feature for FAT filesystem as a separate component, however I don't know when it will be ready. That's why I would like to encourage users to use LittleFS, it's just more modern and optimized for embedded devices. Raising the 2GB limit to two digit number would be great though.

Native MBR partition table support could be worth to consider as well for more flexibility, but right we use f_mkfs from FATFS library which creates MBR/GPT for SD cards.

[huming2207]

We want to address it and are developing a journaling feature for FAT filesystem as a separate component

Hmm yea, I just had a look and I just realised Keil MDK has that.

That's why I would like to encourage users to use LittleFS, it's just more modern and optimized for embedded devices. Raising the 2GB limit to two digit number would be great though.

My colleagues and I read LittleFS's doc and code earlier and we think it might need quite a bit hack to make it bigger than 2GB. But currently for our use cases (mostly logging & statistic stuff) 2GB is enough anyway.

Native MBR partition table support could be worth to consider as well for more flexibility, but right we use f_mkfs from FATFS library which creates MBR/GPT for SD cards.

Yes indeed I think MBR/GPT probably is the right way to go. This could be particularly helpful for HMI devices or surveillance cameras with ESP32-P4, where large storage is necessary for dealing with multimedia content. Users can do something like, a few small-ish LittleFS partitions for the microcontroller to write logs, and one or two bigger FAT partitions for storing videos.