Confirm there is no SDFS/vfat support for files > 4GB? (IDFGH-13768)

JonathanDotCel commented 1 month ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

Fixed some BLE bugs 240826 on ESP32C3 (8ce789b) (v5.3)

Espressif SoC revision.

ESP32 S3 (not chip specific)

Operating System used.

Windows

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

PowerShell

Development Kit.

ESP32S3

Power Supply used.

USB

What is the expected behavior?

Either ftello and fseeko return a 64 bit value, (off_t, _off_t, off64_t, _off64_t, _fpos_t, _fpos64_t, etc) or implementations for e.g. fseeko64 and ftello64

What is the actual behavior?

fseeko64 and ftello64 are not defined any offset type without an explicit "64" in the name is always sizeof(x) == 4 ftello and fseeko always return a 32 bit value (so for files > 4gb you get only the 32bit remainder of the file size)

Steps to reproduce.

Use a simple SDFS demo.
fopen() a >4gb file
fseek() to the end
ask ftell(), ftello(), ftello64() etc

Same thing when stating the file

Debug Logs.

not applicable
just like printf( "...", sizeof(off_t) ); etc
and uint64_t size = ftell();

More Information.

FATFS directly supports 64 bit operations, but you take a performance hit for going that route.

Is there any way to get 64 bit support through the POSIX/VFS layer? Failing that, any pointers on what must be done to get similar speeds out of FATFS directly?

Thanks, J

adokitkat commented 1 month ago

File size limit on FAT32 is 4GB, so I think this would be relevant only if you were to use exFAT. However since exFAT requires a license, so we only officially support up to FAT32 (current VFS implementation supports only 32-bit operations). You could theoretically write your own VFS driver where you use FATFS with 64-bit LBA.

JonathanDotCel commented 1 month ago

File size limit on FAT32 is 4GB, so I think this would be relevant only if you were to use exFAT. However since exFAT requires a license, so we only officially support up to FAT32 (current VFS implementation supports only 32-bit operations). You could theoretically write your own VFS driver where you use FATFS with 64-bit LBA.

Hi, thanks for getting back. I figured the VFS driver might be the case. Are there any performance gains associated with going through the VFS layer (dma, etc), or would that all have to be implemented in the VFS driver?

Thanks, J

pacucha42 commented 1 month ago

Hi @JonathanDotCel, afaik there is nothing performance specific to implement in your VFS version - you just add 64bit versions of required APIs (to both fatfs/vfs and vfs), and compile the code with FF_LBA64 option on, plus you have to deploy the exFAT module (hardwired condition given by the FatFS library). Furthermore, you may need to update fatfs/diskio APIs as they would compile with 64bit arguments (see https://github.com/espressif/esp-idf/blob/master/components/fatfs/src/diskio.h#L33).

This scenario may theoretically work for SD cards, but probably no SPI NOR flash chips provide more than 4GB of space. I'm quite sceptical about any success on this mission, sorry. ESP32* chips are 32bit platform and you may bump into various limitations coming from this construction.

Please, let us know about your progress. Thank you

espressif / esp-idf