Not read file content in external spiffs image (GIT8266O-468)

[vvzvlad]

Environment

• Development Kit:
• IDF version (git rev-parse --short HEAD to get the commit id.): b74255d0
• Development Env: [Make]
• Operating System: [ubuntu]
• Power Supply: [USB]

I'm trying to access the data in spiffs from the firmware, which I made with an external program and I can't do it. 1) Create folder and files:

mkdir data
echo "TST0000" > ./data/tst.tst

2) Create spiffs image:

./mkspiffs --size 458752 --create ./data/ ./spiffs.bin
python spiffsgen.py --size 458752 --block 4096 --page 256 --base_dir ./data/ --output_file ./spiffs.bin

I tried creating both programs. Image settings:

SPIFFS_OBJ_NAME_LEN: 32 SPIFFS_OBJ_META_LEN: 0 SPIFFS_USE_MAGIC: 1 SPIFFS_USE_MAGIC_LENGTH: 1 SPIFFS_ALIGNED_OBJECT_INDEX_TABLES: 0

Size 458752(448k). Version of mkspiffs - mkspiffs ver. 0.2.3-6-g983970e/SPIFFS ver. 0.3.7-5-gf5e26c4 3) Checking the image:

>>./mkspiffs --unpack ./dump_2 spiffs.bin
>>cat dump_2/tst.tst
TST0000

Both images are unpacked correctly. 4) Upload the image to ESP: esptool.py --chip esp8266 --port /dev/ttyUSB0 --baud 115200 --before default_reset --after hard_reset write_flash --flash_mode dio --flash_freq 40m --flash_size 2MB 0x00090000 ./spiffs.bin

I took the address 0x00090000 and the image size (0x00070000=458752=448k) from the bootloader output:

I (88) boot: Partition Table:
I (94) boot: ## Label            Usage          Type ST Offset   Length
I (105) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (117) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (128) boot:  2 factory          factory app      00 00 00010000 00080000
I (140) boot:  3 storage          Unknown data     01 82 00090000 00070000

The loader gets it from partitions.csv:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 512K,
storage,  data, spiffs,  ,        448K,

Checking the count: 0x10000(firmware address)+0x80000(firmware size)=0x00090000(start of spiffs), which is the same as the loader data and the address where I download spiffs.bin.

5)SPIFFS in the firmware correctly defines the beginning of the image (I tried to upload the image to a different address and get an error, there is no error at the correct address, so I conclude that the image is of the correct size and uploaded at the correct address):

I (431) APP: Initializing SPIFFS
I (466) APP: Partition size: total: 414401, used: 3765

I use this code to initialize:

ESP_LOGI(TAG, "Initializing SPIFFS");

esp_vfs_spiffs_conf_t conf = {
    .base_path = "/spiffs",
    .partition_label = NULL,
    .max_files = 5,
    .format_if_mount_failed = true
};
esp_err_t ret = esp_vfs_spiffs_register(&conf);
if (ret != ESP_OK) {
    if (ret == ESP_FAIL) {
        ESP_LOGE(TAG, "Failed to mount or format filesystem");
    } else if (ret == ESP_ERR_NOT_FOUND) {
        ESP_LOGE(TAG, "Failed to find SPIFFS partition");
    } else {
        ESP_LOGE(TAG, "Failed to initialize SPIFFS (%s)", esp_err_to_name(ret));
    }
    return;
}

size_t total = 0, used = 0;
ret = esp_spiffs_info(NULL, &total, &used);
if (ret != ESP_OK) {
    ESP_LOGE(TAG, "Failed to get SPIFFS partition information (%s)", esp_err_to_name(ret));
} else {
    ESP_LOGI(TAG, "Partition size: total: %d, used: %d", total, used);
}

The settings in the firmware are as follows:

>>cat sdkconfig | grep SPIFFS
CONFIG_USING_SPIFFS=y
CONFIG_SPIFFS_MAX_PARTITIONS=3
CONFIG_SPIFFS_CACHE=y
CONFIG_SPIFFS_CACHE_WR=y
# CONFIG_SPIFFS_CACHE_STATS is not set
CONFIG_SPIFFS_PAGE_CHECK=y
CONFIG_SPIFFS_GC_MAX_RUNS=10
# CONFIG_SPIFFS_GC_STATS is not set
CONFIG_SPIFFS_PAGE_SIZE=256
CONFIG_SPIFFS_OBJ_NAME_LEN=32
CONFIG_SPIFFS_USE_MAGIC=y
CONFIG_SPIFFS_USE_MAGIC_LENGTH=y
CONFIG_SPIFFS_META_LENGTH=0
# CONFIG_SPIFFS_DBG is not set
# CONFIG_SPIFFS_API_DBG is not set
# CONFIG_SPIFFS_GC_DBG is not set
# CONFIG_SPIFFS_CACHE_DBG is not set
# CONFIG_SPIFFS_CHECK_DBG is not set
# CONFIG_SPIFFS_TEST_VISUALISATION is not set

Comparing the parameters CONFIG_SPIFFS_META_LENGTH, CONFIG_SPIFFS_PAGE_SIZE, CONFIG_SPIFFS_OBJ_NAME_LEN, CONFIG_SPIFFS_USE_MAGIC, CONFIG_SPIFFS_USE_MAGIC_LENGTH I come to the conclusion that they are the same as programs that create the image, that the firmware has. The size of the image is also correct. Consequently, nothing should interfere with reading the files.

6) SPIFFS correctly determines the presence of the file:

I (469) APP: Reading file tst.tst
I (474) APP: tst.tst size: '8'

I use this code to open the file:

ESP_LOGI(TAG, "Reading file tst.tst");
FILE* f = fopen("/spiffs/tst.tst", "r");
if (f == NULL) {
    ESP_LOGE(TAG, "Failed to open file for reading");
    return;
}
fseek(f, 0, SEEK_END);
uint8_t file_size = ftell(f);
fseek(f, 0, SEEK_SET);
ESP_LOGI(TAG, "tst.tst size: '%d'", file_size);

I tried to open a file that didn't exist, and I failed:

I (582) APP: Reading file no_valid_file.txt.
E (611) APP: Failed to open file for reading

I come to the conclusion that SPIFFS defines the file name correctly.

7) However, reading the file ends unsuccessfully.

I (486) APP: Read from tst.tst: ''.
READ SPIFFS 0 bytes 00 00 00 00 00 00...

I use this code to read the file:

file_size=10;
char line_i[file_size];
uint8_t bytes_read = fread(line_i, 1, file_size, f);
ESP_LOGI(TAG, "Read from tst.tst: '%s'", line_i);
printf("READ SPIFFS %d bytes %02x %02x %02x %02x %02x %02x...\n", bytes_read, line_i[0],line_i[1],line_i[2],line_i[3],line_i[4],line_i[5]);

This behavior on b74255d0844c14153bd78c15298e0202b93eda0e. If I use ee0c4459d6d083cfb9da1ae39e4064c7e9dbf3cd, the picture is different and garbage is read instead of the file contents:

I (456) APP: Read from tst.tst: 'tH'?$@ '.
READ SPIFFS 0 bytes 74 48 48 27 27 40 08 bc...

(Remember, it must be "TST0000")

I don't know what to do.

[nsfilho]

@vvzvlad I faced something like that. The solution is in you compile flags for mkspiffs. I wrote a FAQ in my project, where I show a little about that.

https://github.com/nsfilho/E12AIO3/tree/master/tools

look my sdkconfig.defaults about that compiler flags. OK?

[vvzvlad]

@nsfilho You mean the parameter "SPIFFS_ALIGNED_OBJECT_INDEX_TABLES=1"?

[vvzvlad]

Yes, the "SPIFFS_ALIGNED_OBJECT_INDEX_TABLES=1" parameter makes it work:

I (365) example: Initializing SPIFFS
I (378) example: Partition size: total: 37901, used: 502
I (381) example: Reading file
I (388) example: tst.tst poz: '0'
I (394) example: Read from file: 'TST
'
I (400) example: SPIFFS unmounted

[nsfilho]

Nice! Now, you can close the issue :))

[nsfilho]

Please @vvzvlad don't forget to close the issue. If we help espressif in this way, probably the engineers from company can work on real bugs....

[XiaoSenLuo]

I also faced that, and it soluts my trouble, thanks.

[colesnicov]

Yes, the "SPIFFS_ALIGNED_OBJECT_INDEX_TABLES=1" parameter makes it work:

I (365) example: Initializing SPIFFS
I (378) example: Partition size: total: 37901, used: 502
I (381) example: Reading file
I (388) example: tst.tst poz: '0'
I (394) example: Read from file: 'TST
'
I (400) example: SPIFFS unmounted

It is work for me too.

[pauloasherring]

+1, that solved my issue.

[nsfilho]

Very nice! We need share ideas and things around this community. Because ESP32 is reality, but some of us have a lot of ESP8266 solutions running :)

Sometimes ESP8266-RTOS appears forgeted in face of ESP32. But some hardware as we produce (in my case, for my house automation) is 100% based on this, and I can't change some of them.

Espressif, please you need have a long term support and R&D for your legacy solutions. Because in more 3 or 4 years, ESP32 will old too. What happens at this time? An entirelly eco-systems depends of you.

We, your community is here. We can help, we can participate on that. If you assign a technical leader for this project (from Espressif, what have access to all info) to conduce us, we will help you to maintain this solution.

In my personal case, I depend of this. It is not a question of if I want to contribute, I really need this project alive. :)

Thank you all, NS.