FAT FS: unable to read file with long name which contains underscores (IDFGH-10008)

narangmayank commented 1 year ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

5.0

Operating System used.

Windows

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

CMD

Development Kit.

ESP32

Power Supply used.

USB

What is the expected behavior?

It should read the content of the file

What is the actual behavior?

Error while opening the file.

Steps to reproduce.

Try the below code

Debug Logs.

I (0) cpu_start: App cpu up.
I (235) cpu_start: Pro cpu start user code
I (235) cpu_start: cpu freq: 160000000 Hz
I (235) cpu_start: Application information:
I (240) cpu_start: Project name:     fatfsgen
I (245) cpu_start: App version:      v5.0-dirty
I (250) cpu_start: Compile time:     Apr 28 2023 00:34:35
I (256) cpu_start: ELF file SHA256:  0394a7ebdab63cd9...
I (262) cpu_start: ESP-IDF:          v5.0-dirty
I (268) heap_init: Initializing. RAM available for dynamic allocation:
I (275) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (281) heap_init: At 3FFB2FC0 len 0002D040 (180 KiB): DRAM
I (287) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (293) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (300) heap_init: At 4008BAA8 len 00014558 (81 KiB): IRAM
I (307) spi_flash: detected chip: generic
I (311) spi_flash: flash io: dio
I (315) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (325) example: Mounting FAT filesystem
I (335) example: FATFS Mounted.
I (335) example: Reading file : /spiflash/device_config.json
E (345) example: Failed to open device config file for reading
I (345) example: Done

More Information.

I've already enable the long file support in the FAT component config and with this I'm able to read the file which has length more than 8 characters as specified in the docs but it is not allowing me to read if the file extension has more tha 3 characters.

any help would be appreciated. Thanks in advance.

void read_file()
{
    const char* path = "/spiflash/device_config.json";
    ESP_LOGI(TAG, "Reading file : %s", path);

    FILE *file = fopen(path, "r");

    if (file == NULL) 
    {
        ESP_LOGE(TAG, "Failed to open device config file for reading");
        return;
    }

    fseek(file, 0, SEEK_END);
    int file_length = ftell(file);
    fseek(file, 0, SEEK_SET);

    if(file_length <= 0)
    {
        ESP_LOGE(TAG, "Failed to get device config file size");
        return;
    }

    // dynamically allocate a char array to store the file contents
    bytebeam_device_config_data = malloc(sizeof(char) * (file_length + 1));

    // if memory allocation fails just log the failure to serial and return :)
    if(bytebeam_device_config_data == NULL)
    {
        ESP_LOGE(TAG, "Failed to allocate the memory for device config file");
        return;
    }

    int temp_c;
    int loop_var = 0;

    while ((temp_c = fgetc(file)) != EOF)
    {
        bytebeam_device_config_data[loop_var] = temp_c;
        loop_var++;
    }

    bytebeam_device_config_data[loop_var] = '\0';

    ESP_LOGI(TAG, "data : %s\n", bytebeam_device_config_data);

    fclose(file);
}

void app_main(void)
{
    ESP_LOGI(TAG, "Mounting FAT filesystem");
    // To mount device we need name of device partition, define base_path
    // and allow format partition in case if it is new one and was not formatted before
    const esp_vfs_fat_mount_config_t mount_config = {
            .max_files = 4,
            .format_if_mount_failed = false,
            .allocation_unit_size = CONFIG_WL_SECTOR_SIZE
    };
    esp_err_t err;

    // if (EXAMPLE_FATFS_MODE_READ_ONLY){
    //     err = esp_vfs_fat_spiflash_mount_ro(base_path, "storage", &mount_config);
    // } else {
    //     err = esp_vfs_fat_spiflash_mount_rw_wl(base_path, "storage", &mount_config, &s_wl_handle);
    // }

    err = esp_vfs_fat_spiflash_mount_ro(base_path, "storage", &mount_config);

    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to mount FATFS (%s)", esp_err_to_name(err));
        return;
    } else {
        ESP_LOGI(TAG, "FATFS Mounted.");
    }

    read_file();
    ESP_LOGI(TAG, "Done");
}

igrr commented 1 year ago

I have searched the issue tracker for a similar issue and not found a similar issue.

I'm not sure if this didn't come up when you have searched for similar issues, please check https://github.com/espressif/esp-idf/issues/6138, https://github.com/espressif/esp-idf/issues/6160

I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.

Quoting https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/fatfs.html,

The filesystem uses 8.3 filenames format (SFN) by default. If you need to use long filenames (LFN), enable the CONFIG_FATFS_LONG_FILENAMES option.

narangmayank commented 1 year ago

Hi @igrr, I've came across LFN option for FATFS during my search and I already tried this but not able to get it working. Issue is poping up only when file extension length grow after 3. For eg. I can read device_config.txt but not device_config.json.

Here are my FATFS conf,

#
# FAT Filesystem support
#
CONFIG_FATFS_VOLUME_COUNT=2
# CONFIG_FATFS_SECTOR_512 is not set
# CONFIG_FATFS_SECTOR_1024 is not set
# CONFIG_FATFS_SECTOR_2048 is not set
CONFIG_FATFS_SECTOR_4096=y
CONFIG_FATFS_SECTORS_PER_CLUSTER_1=y
# CONFIG_FATFS_SECTORS_PER_CLUSTER_2 is not set
# CONFIG_FATFS_SECTORS_PER_CLUSTER_4 is not set
# CONFIG_FATFS_SECTORS_PER_CLUSTER_8 is not set
# CONFIG_FATFS_SECTORS_PER_CLUSTER_16 is not set
# CONFIG_FATFS_SECTORS_PER_CLUSTER_32 is not set
# CONFIG_FATFS_SECTORS_PER_CLUSTER_64 is not set
# CONFIG_FATFS_SECTORS_PER_CLUSTER_128 is not set
# CONFIG_FATFS_CODEPAGE_DYNAMIC is not set
CONFIG_FATFS_CODEPAGE_437=y
# CONFIG_FATFS_CODEPAGE_720 is not set
# CONFIG_FATFS_CODEPAGE_737 is not set
# CONFIG_FATFS_CODEPAGE_771 is not set
# CONFIG_FATFS_CODEPAGE_775 is not set
# CONFIG_FATFS_CODEPAGE_850 is not set
# CONFIG_FATFS_CODEPAGE_852 is not set
# CONFIG_FATFS_CODEPAGE_855 is not set
# CONFIG_FATFS_CODEPAGE_857 is not set
# CONFIG_FATFS_CODEPAGE_860 is not set
# CONFIG_FATFS_CODEPAGE_861 is not set
# CONFIG_FATFS_CODEPAGE_862 is not set
# CONFIG_FATFS_CODEPAGE_863 is not set
# CONFIG_FATFS_CODEPAGE_864 is not set
# CONFIG_FATFS_CODEPAGE_865 is not set
# CONFIG_FATFS_CODEPAGE_866 is not set
# CONFIG_FATFS_CODEPAGE_869 is not set
# CONFIG_FATFS_CODEPAGE_932 is not set
# CONFIG_FATFS_CODEPAGE_936 is not set
# CONFIG_FATFS_CODEPAGE_949 is not set
# CONFIG_FATFS_CODEPAGE_950 is not set
CONFIG_FATFS_AUTO_TYPE=y
# CONFIG_FATFS_FAT12 is not set
# CONFIG_FATFS_FAT16 is not set
CONFIG_FATFS_CODEPAGE=437
# CONFIG_FATFS_LFN_NONE is not set
# CONFIG_FATFS_LFN_HEAP is not set
CONFIG_FATFS_LFN_STACK=y
CONFIG_FATFS_MAX_LFN=255
CONFIG_FATFS_API_ENCODING_ANSI_OEM=y
# CONFIG_FATFS_API_ENCODING_UTF_8 is not set
CONFIG_FATFS_FS_LOCK=0
CONFIG_FATFS_TIMEOUT_MS=10000
CONFIG_FATFS_PER_FILE_CACHE=y
# CONFIG_FATFS_USE_FASTSEEK is not set
# end of FAT Filesystem support

igrr commented 1 year ago

I see now, could you please describe how you are generating and flashing the filesystem? Are you following what fatfsgen example does, or doing it manually instead?

As a troubleshooting step, you can try to list all the file names in FAT partition using opendir/readdir functions. Perhaps the file name was not correctly written into the FAT image when generating it on the host?

narangmayank commented 1 year ago

I used this to generate the FAT partition during the build.

fatfs_create_rawflash_image(storage ../config_data FLASH_IN_PROJECT)

adokitkat commented 1 year ago

Short File Name (SFN) respects 8.3 format name which means up to 8 character file name and 3 character file extension. It seems like Long File Name (LFN) only extends the file name length limitation up to 255 characters, not the file extension length limitation. However I think the file extension only gets truncated to the first 3 characters so you should be able to access file.json as file.jso.

narangmayank commented 1 year ago

@igrr @adokitkat I debugged this guy with opndir/readdir and found that the file name itself is changing. Yes you're right the extension I got is .jso so ya it can be readable if file name is not changed.

could be please help me to debug this further?

Code

void list_content()
{
    DIR *dp;
    struct dirent *ep;

    dp = opendir ("/spiflash");

    if (dp != NULL) {
        ep = readdir(dp);
        while (ep != NULL) {
            ESP_LOGI(TAG, "name : %s", ep->d_name);
            ep = readdir(dp);
        }
        closedir(dp);
    } else {
        ESP_LOGI(TAG, "Couldn't open the directory");
    }
}

void read_file()
{
    const char* path = "/spiflash/device_config.json";
    ESP_LOGI(TAG, "Reading file : %s", path);

    FILE *file = fopen(path, "r");

    if (file == NULL) 
    {
        ESP_LOGE(TAG, "Failed to open device config file for reading");
        return;
    }

    fseek(file, 0, SEEK_END);
    int file_length = ftell(file);
    fseek(file, 0, SEEK_SET);

    if(file_length <= 0)
    {
        ESP_LOGE(TAG, "Failed to get device config file size");
        return;
    }

    // dynamically allocate a char array to store the file contents
    bytebeam_device_config_data = malloc(sizeof(char) * (file_length + 1));

    // if memory allocation fails just log the failure to serial and return :)
    if(bytebeam_device_config_data == NULL)
    {
        ESP_LOGE(TAG, "Failed to allocate the memory for device config file");
        return;
    }

    int temp_c;
    int loop_var = 0;

    while ((temp_c = fgetc(file)) != EOF)
    {
        bytebeam_device_config_data[loop_var] = temp_c;
        loop_var++;
    }

    bytebeam_device_config_data[loop_var] = '\0';

    ESP_LOGI(TAG, "data : %s\n", bytebeam_device_config_data);

    fclose(file);
}

void app_main(void)
{
    ESP_LOGI(TAG, "Mounting FAT filesystem");
    // To mount device we need name of device partition, define base_path
    // and allow format partition in case if it is new one and was not formatted before
    const esp_vfs_fat_mount_config_t mount_config = {
            .max_files = 4,
            .format_if_mount_failed = false,
            .allocation_unit_size = CONFIG_WL_SECTOR_SIZE
    };
    esp_err_t err;

    // if (EXAMPLE_FATFS_MODE_READ_ONLY){
    //     err = esp_vfs_fat_spiflash_mount_ro(base_path, "storage", &mount_config);
    // } else {
    //     err = esp_vfs_fat_spiflash_mount_rw_wl(base_path, "storage", &mount_config, &s_wl_handle);
    // }

    err = esp_vfs_fat_spiflash_mount_ro(base_path, "storage", &mount_config);

    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to mount FATFS (%s)", esp_err_to_name(err));
        return;
    } else {
        ESP_LOGI(TAG, "FATFS Mounted.");
    }

    list_content();
    read_file();

    ESP_LOGI(TAG, "Done");
}

Serial Logs

I (0) cpu_start: App cpu up.
I (235) cpu_start: Pro cpu start user code
I (235) cpu_start: cpu freq: 160000000 Hz
I (236) cpu_start: Application information:
I (240) cpu_start: Project name:     fatfsgen
I (245) cpu_start: App version:      v5.0-dirty
I (251) cpu_start: Compile time:     May  2 2023 20:07:48
I (257) cpu_start: ELF file SHA256:  85aa6f29195c8eb7...
I (263) cpu_start: ESP-IDF:          v5.0-dirty
I (268) heap_init: Initializing. RAM available for dynamic allocation:
I (275) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (281) heap_init: At 3FFB2FC0 len 0002D040 (180 KiB): DRAM
I (287) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (294) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (300) heap_init: At 4008BAA8 len 00014558 (81 KiB): IRAM
I (308) spi_flash: detected chip: generic
I (311) spi_flash: flash io: dio
I (316) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (326) example: Mounting FAT filesystem
I (336) example: FATFS Mounted.
I (336) example: name : DEVICE~1.JSO
I (336) example: name : test.txt
I (346) example: Reading file : /spiflash/device_config.json
E (346) example: Failed to open device config file for reading
I (356) example: Done

igrr commented 1 year ago

@mayankbytebeam It looks like your fatfs image wasn't re-generated when you have enabled the LFN option.

I see no problem with extensions longer than 3 characters in examples/storage/fatfsgen:

open menuconfig
enable CONFIG_FATFS_LFN_HEAP
enable CONFIG_EXAMPLE_FATFS_MODE_READ_ONLY
rename fatfs_long_name_image/sublongnames/testlongfilenames.txt to fatfs_long_name_image/sublongnames/testlongfilenames.text
modify examples/storage/fatfsgen/main/fatfsgen_example_main.c, changing sublongnames/testlongfilenames.txt to sublongnames/testlongfilenames.text
idf.py flash monitor

and the file is correctly read:

I (362) example: Reading file
I (362) example: The file '/spiflash/sublongnames/testlongfilenames.text' was modified at date: 2023-05-02
I (372) example: Read from file: 'this is test; long name it has'

So the issue on your side could possibly be:

fatfs image (build/storage.bin) was not re-generated after changing sdkconfig?
- try deleting the .bin file and then building again, to make sure it got generated again
fatfs image was generated, but wasn't flashed to the board
- unlikely, since you have FLASH_IN_PROJECT correctly set in CMake; still, you can read back the image using esptool.py read_flash and then compare the storage .bin file in the build directory with the one you got

adokitkat commented 1 year ago

I see, thanks for the explanation. The FATFS docs could have mentioned it there...

narangmayank commented 1 year ago

@igrr Thanks for coming up.

I was checking the fat file format and get to know that it doesn't allow _ in the file name. I'm not sure if you notice this or not I was trying to write/read device_config.json file. So as per their logic file name is changing to DEVICE~1.JSO. You can read up more here

Solution: I'm now using deviceconfig.json file name and everything looks good.

@igrr @adokitkat Thanks for your time, I really appreciate it.

igrr commented 1 year ago

That's a very interesting observation @mayankbytebeam! I can confirm this behavior.

I don't think this is a limitation of FAT filesystem format, though. After modifying the long filename in the example from testlongfilenames.txt to test_long_filenames.text and generating the new disk image, the image can be mounted on the PC, and the filenames are as expected (i.e. with underscores):

$ hdiutil attach -blocksize 4096 -imagekey diskimage-class=CRawDiskImage -nomount build/storage.bin
/dev/disk6
$ mkdir test
$ mount -o ro -t msdos /dev/disk6 $PWD/test
Executing: /usr/bin/kmutil load -p /System/Library/Extensions/msdosfs.kext
$ ls -1R test
hellolongname.txt*
sublongnames/

test/sublongnames:
test_long_filenames.text*

(this is on macOS; commands to mount a raw FAT image on other OSes are different)

Since on the desktop we can see the correct long file name (with underscores) after mounting the image, the issue could be on the ESP side, in FATFS library. We'll investigate this further, but for now you have a work-around :)

I'll also rename the issue so that other may find it easier.

narangmayank commented 1 year ago

Since you confirmed the fatfs supports underscore in the file name then yes probably the issue is coming from the esp fatfs library itself.

@igrr Thanks for the insights :)

adokitkat commented 1 year ago

Hi @mayankbytebeam, I've created an example project trying to simulate your problem, however any file name works for me (/spiflash/deviceconfig.json, /spiflash/device-config.json, /spiflash/device_config.json, /spiflash/hello_world/dev_config.json). You can download it here:

fatfs_gen_lfn.zip

Take a look at main/CMakeLists.txt file - fatfs_create_spiflash_image() there uses fatfsgen.py and creates a FATFS image from fatfs_img folder and is flashed to your ESP32 during idf.py flash. Can you try it please? Before flashing do idf.py erase-flash to your ESP32.

narangmayank commented 1 year ago

Hi @adokitkat, I was using fatfs read only mode and spiffs_create_partition_image(storage ../config_data FLASH_IN_PROJECT) to generate the fatfs image in my project.

adokitkat commented 1 year ago

Wait are you using FATFS or SPIFFS? spiffs_create_partition_image is for a SPIFFS partition generation, not a FATFS partition generation.

You should use either:

fatfs_create_spiflash_image with esp_vfs_fat_spiflash_mount_rw_wl (WL) or
fatfs_create_rawflash_image with esp_vfs_fat_spiflash_mount_ro (without WL)

Both do work for me.

narangmayank commented 1 year ago

Yes Sorry, that was a typo from my side. I'm using fatfs_create_rawflash_image with esp_vfs_fat_spiflash_mount_ro. can you please share the working example on this.

narangmayank commented 1 year ago

@adokitkat I just tried your example app and here are the logs. I just want to know whether you checked on idf 5.x release or 4.x release.

I (320) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (330) example: Mounting FAT filesystem
I (340) example: FATFS Mounted.
I (340) example: Reading file : /spiflash/deviceconfig.json
I (350) example: data : {"menu": {
    "id": "file2",
    "value": "File",
    "popup": {
      "menuitem": [
        {"value": "New", "onclick": "CreateDoc()"},
        {"value": "Open", "onclick": "OpenDoc()"},
        {"value": "Save", "onclick": "SaveDoc()"}
      ]
    }
  }}

I (370) example: Reading file : /spiflash/device-config.json
E (380) example: Failed to open device config file for reading
I (380) example: Reading file : /spiflash/device_config.json
E (390) example: Failed to open device config file for reading
I (400) example: Reading file : /spiflash/hello_world/dev_config.json
E (400) example: Failed to open device config file for reading
I (560) example: Done

adokitkat commented 1 year ago

Sorry for the wait. Yes, you're right, it doesn't work on release/v5.0. Please try release/v5.1 or master branch, it works properly there for me.

espressif / esp-idf