Open kmihaylov opened 5 years ago
@kmihaylov You've highlighted the problem very accurately! The logic in spiffs_mount_internal()
(spiffs_sming.c
line 114) is too simplistic -
uint32_t dat;
bool writeFirst = false;
flashmem_read(&dat, cfg->phys_addr, 4);
//debugf("%X", dat);
if (dat == UINT32_MAX)
{
debugf("First init file system");
spiffs_format_internal(cfg);
writeFirst = true;
}
The volume gets wiped if SPIFFS sector #0 is in the erased state when the device reboots, which is a perfectly legitimate state for SPIFFS to be in.
Here's a better way:
// Call SPIFFS_mount() with appropriate parameters
int res = do_mount();
// Check result
if(res < 0) {
// Mount failed, format the volume then retry
format();
res = do_mount();
}
// Final result of mount operation
return res;
There's also a volume checking function in SPIFFS, which is time-consuming but could be integrated like this:
int res = do_mount();
if(res < 0) {
// Mount failed, perform a volume check
check_fs();
res = do_mount();
if(res < 0) {
format();
res = do_mount();
}
}
return res; // Final result of mount operation
You can find some example code for the above here
I did as you suggested:
Now it successfully mounts the fs every time (even if I flashed the device with DISABLE_SPIFFS = 1). The files are written to the FS, but I'm curious how i works if it wasn't inited. Also the problem of unnecessary inited FS disappeared and the files seem to contain the data they should contain.
SPIFFS magically inits the FS on the fly or something else happens that I don't understand yet.
Happy that's working for you! The DISABLE_SPIFFS
only controls whether an image is built and uploaded during flash.
SPIFFS magically inits the FS on the fly
Pretty much! It doesn't erase sectors unless it needs the space. Have you tested the check_fs()
part yet? If you wanted to do some stress testing, try filling the flash with random data first. That should fail the first mount operation and force it to do a check_fs()
.
It mounts the FS despite it was prefilled with random data.
/opt/esp-open-sdk/esptool/esptool.py -p /dev/ttyUSB0 -b 115200 write_flash -ff 40m -fm qio -fs 32m \
0x00000 out/firmware/rboot.bin \
0x02000 out/firmware/rom0.bin \
0x100000 random.bin && python -m serial.tools.miniterm /dev/ttyUSB0 115200
The mount() function is called 10 seconds after boot to ensure that the debug messages refer to the first mount of the random-data space (I had fear that it formats the FS too quickly).
fileSetContent however now returns open errno -10001
.
FS is unusable.
Flashing zeros to the SPIFFS area enables successful fileSetContent() operation, but the FS check returns "-10024"
$ /opt/esp-open-sdk/esptool/esptool.py -p /dev/ttyUSB0 -b 115200 write_flash -ff 40m -fm qio -fs 32m 0x100000 zero.bin
WARNING: Flash size arguments in megabits like '32m' are deprecated.
Please use the equivalent size '4MB'.
Megabit arguments may be removed in a future release.
esptool.py v2.6-beta1
Serial port /dev/ttyUSB0
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
MAC: bc:dd:c2:91:cf:b1
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Compressed 65536 bytes to 84...
Wrote 65536 bytes (84 compressed) at 0x00100000 in 0.0 seconds (effective 44188.1 kbit/s)...
Hash of data verified.
Leaving...
Hard resetting via RTS pin...
$ python -m serial.tools.miniterm /dev/ttyUSB0 115200
--- Miniterm on /dev/ttyUSB0 115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
--- DTR inactive ---
--- RTS inactive ---
{l␀$ܞ|␀�l�|␂␌␄␌�␄l�␌#|����␒�r�B�␌#�p~�o�lon���␌B␜p��cl ␃{lp�N�␐␂␌␄�␌$␌␄␌␄␌␌B␌o��ol�␄��␄b��oN�␃d��␀$`␂�2␒oN␌$`␃␎␃or�ےn␄␌2�� ␂p�n�␐␂␌␄{�ܜ��␌␄␌␄b␄N�|␂����␌"��no�␃␌␄l ␂�␒␛ond�␜{l␏␂n{���o␌␌b��`␃␎rےn␄␌B�� ␂r$�#�no���p|␄�|�␄lB�|Br␃N��␓B�#��o␀��␛␒no"p␡�n␎$␜�N�␂n�o~␒#␄␌␄�␄l$pc��`␂␜�o�␂nl$n��c␄␌␌$�$l␎$␜�o�␂N�␒����>␀␌␄��␌$l$~r$�#�no��$�␛Nno␞␀�␒n���N�␂n␎$rd␌�on�#�n��|␡�o␄�#␏l��␃�␓�o�r��o|�␌l␄$ll ␂��r�l��: 0079839 Commandhandlercommand status registered
80249 Commandhandlercommand echo registered
80548 Commandhandlercommand help registered
80850 Commandhandlercommand debugon registered
84834 Commandhandlercommand debugoff registered
89011 Commandhandlercommand command registered
Welcome to the Stream Command executor
96380 hwsDelegateDemo instantiated, waiting for data
mode : softAP(be:dd:c2:91:cf:b1)
add if1
dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw:192.168.4.1)
bcn 100
10101001 FS check res: -10024
10101225 trying to mount spiffs at 0x00100000, length 65536
10101420 fs.start: size:64 Kb, offset:0x100000
10103543 mount res: 0
save 1
Full cmd is: save 1
Cmd1: save, cmd2: 1
27644332 api_spiffs_erase
27688299 api_spiffs_erase
27712270 api_spiffs_erase
27734290 api_spiffs_erase
27757976 api_spiffs_erase
27779983 api_spiffs_erase
27805229 api_spiffs_erase
27825398 api_spiffs_erase
27847614 Bytes written to fs: 10
--- RTS active ---
--- RTS inactive ---
rl␀l��|␀�l�|␃␌␄␌�␌l�␌#|����␒�r�#�␌#�p␡�n�loN���␌c␜p�$r$rlp�o�␐␃␄␌�␄l␄␌␌␄␌␄b␌N�|␂l�␄��␄b��no�␃l��␀$`␃�␓2Nn␄$`␂␎␃or�ےn␄␌2�� ␂p�N�␐␂␌␌r�����␄␌␄␌c␄o�|␃���␌␄b��oN�␃␄␌d ␃�␛2No$�␜rl␏␂n{���N␌␄c��␎d␎sےn␄␌#�� ␂r$�#�no���p|␌�b␜␀lc�␛op`␛b�␒b�b��n␀��o�b�Nbp~�o␏l␜�n�␂n�o~␒#␄␌␌�␜␀$lp"�� ␃␜�n�␃o$lo���␄␌␄l��~� ␃␜�n�␂n�␛���␐b␌␄␄��␄l$l␡{l�b�on�␀�l�␛oNo>␀� B�␃��o�␃N␎lrl␌�on��|�␂�|␡�n␌�b␎$��␂�␓�N�|␛r��o|�␌d␄ll$`␃��s�p���: 0080166 Commandhandlercommand status registered
80576 Commandhandlercommand echo registered
80875 Commandhandlercommand help registered
81178 Commandhandlercommand debugon registered
85161 Commandhandlercommand debugoff registered
89338 Commandhandlercommand command registered
Welcome to the Stream Command executor
96707 hwsDelegateDemo instantiated, waiting for data
mode : softAP(be:dd:c2:91:cf:b1)
add if1
dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw:192.168.4.1)
bcn 100
10101328 FS check res: -10024
10101552 trying to mount spiffs at 0x00100000, length 65536
10101747 fs.start: size:64 Kb, offset:0x100000
10103860 mount res: 0
save 1
Full cmd is: save 1
Cmd1: save, cmd2: 1
26884600 api_spiffs_erase
26906674 api_spiffs_erase
26931661 Bytes written to fs: 10
save 1
Full cmd is: save 1
Cmd1: save, cmd2: 1
30227023 Bytes written to fs: 10
--- RTS active ---
--- RTS inactive ---
s$␀l��|␀�l�|␃␌␄␌�␄l�␌c|��␂�p␃�s��␟�␄"��no�loN���␌c␜p�$r$rlp�n�␐␂␌␌�␌l␄␌␄␌␌␄b␄N�|␂l�␄��␄"��no�␂l��␀l ␂�␛␒no␄l`␂␎␃or�ےn␌␄␛�`␂p�o�␐␃␄␌{�ܜ��␌␄␄␌#␌o�|␃����␄b��oo�␃␌␄l ␂�␒␛oNl�l`␂␎␃or�ےn␌␄c��␎l␎rےn␌␄c��␎drl�b�oN�␀��p|␄�|�␄lc�␛op`␛b�2b�B��o␀��n�#�ncp~�o␏l␜�N�␂n�n␡␛b␌␌␄�␄l$pb��`␃␜�n�␂ndlo��c␌␄␌l�ld␏l␜�N�␂n�␒����>␀␌␌�␌dll~rl�b�on�␀�l����on␟␀�␛o܌�n�␃o␏lr$␄�oo��|�܌�n�␀␌�#␎l��␂�p␃�N천p��o|�␌l␄$ll ␂��r�l��: 0080381 Commandhandlercommand status registered
80791 Commandhandlercommand echo registered
81090 Commandhandlercommand help registered
81392 Commandhandlercommand debugon registered
85377 Commandhandlercommand debugoff registered
89553 Commandhandlercommand command registered
Welcome to the Stream Command executor
96922 hwsDelegateDemo instantiated, waiting for data
mode : softAP(be:dd:c2:91:cf:b1)
add if1
dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw:192.168.4.1)
bcn 100
10101543 FS check res: -10024
10101766 trying to mount spiffs at 0x00100000, length 65536
10101962 fs.start: size:64 Kb, offset:0x100000
10104072 mount res: 0
save 1
Full cmd is: save 1
Cmd1: save, cmd2: 1
26142280 Bytes written to fs: 10
save 100
Full cmd is: save 100
Cmd1: save, cmd2: 100
32530623 Bytes written to fs: 10
32538193 Bytes written to fs: 20
32545746 Bytes written to fs: 30
32553297 Bytes written to fs: 40
32560841 Bytes written to fs: 50
32568382 Bytes written to fs: 60
32575903 Bytes written to fs: 70
32583418 Bytes written to fs: 80
32590923 Bytes written to fs: 90
32598428 Bytes written to fs: 100
32605916 Bytes written to fs: 110
32613453 Bytes written to fs: 120
32619477 api_spiffs_erase
32639630 api_spiffs_erase
32662821 Bytes written to fs: 130
32670502 Bytes written to fs: 140
32678071 Bytes written to fs: 150
32685628 Bytes written to fs: 160
32693185 Bytes written to fs: 170
32700725 Bytes written to fs: 180
32708258 Bytes written to fs: 190
32715782 Bytes written to fs: 200
32723306 Bytes written to fs: 210
32730809 Bytes written to fs: 220
32738312 Bytes written to fs: 230
32745799 Bytes written to fs: 240
32753283 Bytes written to fs: 250
32760754 Bytes written to fs: 260
32768222 Bytes written to fs: 270
32775211 api_spiffs_erase
32797328 api_spiffs_erase
32821471 Bytes written to fs: 280
32828978 Bytes written to fs: 290
32836509 Bytes written to fs: 300
32844041 Bytes written to fs: 310
32851552 Bytes written to fs: 320
32859058 Bytes written to fs: 330
32866556 Bytes written to fs: 340
32874056 Bytes written to fs: 350
32881533 Bytes written to fs: 360
32889005 Bytes written to fs: 370
32896465 Bytes written to fs: 380
32903923 Bytes written to fs: 390
32911366 Bytes written to fs: 400
32918805 Bytes written to fs: 410
32926230 Bytes written to fs: 420
32933701 Bytes written to fs: 430
32939638 api_spiffs_erase
32959837 api_spiffs_erase
32984891 Bytes written to fs: 440
32992478 Bytes written to fs: 450
32999960 Bytes written to fs: 460
33007429 Bytes written to fs: 470
33014889 Bytes written to fs: 480
33022340 Bytes written to fs: 490
33029792 Bytes written to fs: 500
33037223 Bytes written to fs: 510
33044652 Bytes written to fs: 520
33052074 Bytes written to fs: 530
33059480 Bytes written to fs: 540
33066878 Bytes written to fs: 550
33074268 Bytes written to fs: 560
33081644 Bytes written to fs: 570
33089021 Bytes written to fs: 580
33095965 api_spiffs_erase
33121164 api_spiffs_erase
33150843 Bytes written to fs: 590
33158213 Bytes written to fs: 600
33165800 Bytes written to fs: 610
33173382 Bytes written to fs: 620
33180954 Bytes written to fs: 630
33188526 Bytes written to fs: 640
33196079 Bytes written to fs: 650
33203628 Bytes written to fs: 660
33211169 Bytes written to fs: 670
33218702 Bytes written to fs: 680
33226221 Bytes written to fs: 690
33233737 Bytes written to fs: 700
33241235 Bytes written to fs: 710
33248736 Bytes written to fs: 720
33256221 Bytes written to fs: 730
33263748 Bytes written to fs: 740
33269893 api_spiffs_erase
33294605 api_spiffs_erase
33322252 Bytes written to fs: 750
33329997 Bytes written to fs: 760
33337575 Bytes written to fs: 770
33345146 Bytes written to fs: 780
33352708 Bytes written to fs: 790
33360261 Bytes written to fs: 800
33367802 Bytes written to fs: 810
33375335 Bytes written to fs: 820
33382864 Bytes written to fs: 830
33390379 Bytes written to fs: 840
33397884 Bytes written to fs: 850
33405382 Bytes written to fs: 860
33412880 Bytes written to fs: 870
33420358 Bytes written to fs: 880
33427830 Bytes written to fs: 890
33434934 api_spiffs_erase
33459593 api_spiffs_erase
33486327 Bytes written to fs: 900
33493897 Bytes written to fs: 910
33501487 Bytes written to fs: 920
33509068 Bytes written to fs: 930
33516636 Bytes written to fs: 940
33524202 Bytes written to fs: 950
33531747 Bytes written to fs: 960
33539296 Bytes written to fs: 970
33546829 Bytes written to fs: 980
33554356 Bytes written to fs: 990
33561877 Bytes written to fs: 1000
--- RTS active ---
--- RTS inactive ---
rd␀lܟ|␀�$�|␂␄␌␄�␌d�␌#|��␂�2�{��␟�␄b��no�loN���␌c␜p��$r$rlp�n�␐␂␌␄�␌$␌␄␄␌␄␌c␄n�|␂l�␌��␄b��no�␃l��␀$`␃�␒␛on␌$`␃␏␂n{���o␌␌␒� ␃p�n�␐␂␌␌r�����␄␄␌␄b␌N��o���␄␄b��oN�␃␌␄l ␂�2␒ool�$`␃␏␂n{���o␌␌b�� ␃␏r��o␌␌b�� ␃{$�#�no�����b�"␜␀$b�␒np ␒c�␒#��|��n␀���2␒no"p␡�N␎d␜�o�␃o���B␄␌␄�␄ldpb��`␃␜�n�␂o$lo���␄␄␌$�d$␎l␜�n�␃o�␛���␐B␄␌␄��␌$ld~rl�b�oN�␀�l����oN␟␀�␓o���N�␂N␎drl␌�Nn��|�␂�|␡�n␌�b␎$��␃�␒�n�s��N|�␌d␄lld ␃�n�l��: 0080305 Commandhandlercommand status registered
80716 Commandhandlercommand echo registered
81015 Commandhandlercommand help registered
81317 Commandhandlercommand debugon registered
85301 Commandhandlercommand debugoff registered
89477 Commandhandlercommand command registered
Welcome to the Stream Command executor
96846 hwsDelegateDemo instantiated, waiting for data
mode : softAP(be:dd:c2:91:cf:b1)
add if1
dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw:192.168.4.1)
bcn 100
10101468 FS check res: -10024
10101691 trying to mount spiffs at 0x00100000, length 65536
10101886 fs.start: size:64 Kb, offset:0x100000
10103999 mount res: 0
This is probably an unrealistic test, for the flash to get in that state would probably indicate other problems so perhaps not realistic. For more robust operation, try setting SPIFFS_USE_MAGIC=1
in spiffs_config.h
- further details here.
@mikee47 respect for your guidance. It's very helpful.
I put
#define SPIFFS_USE_MAGIC (1)
and
#define SPIFFS_USE_MAGIC_LENGTH 1
And now after make flashinit
it works almost as expected:
I did a dump of the formatted partition and it is 0xFF everywhere, obviously there are no magic numbers.
So far, so good.
I thought to put some data (in order for the FS to put some magic numbers inside), but since the FS couldn't be mount this option is ignored.
I read spiffs_format_internal
in spiffs_sming.c and I don't see the function to write some magic numbers during format.
So I guess it only wipes the flash
while( sect_first <= sect_last )
{
if(flashmem_erase_sector( sect_first++ ))
{
int percent = ++cur * 100 / total;
if (percent > last)
debugf("%d%%", percent);
last = percent;
}
else
{
ETS_INTR_UNLOCK();
return false;
}
}
UPDATE: I just found the correct procedure for initializing SPIFFS with magics: https://github.com/pellepl/spiffs/wiki/Using-spiffs
If SPIFFS_USE_MAGIC is enabled, mounting procedure will look for magic. Recommended formatting procedure is as point two above. Recapped:
Call SPIFFS_mount
If SPIFFS_mount fails with SPIFFS_ERR_NOT_A_FS, keep going. Otherwise, call SPIFFS_unmount
Call SPIFFS_format
Call SPIFFS_mount again.
Well, I tried to modify the code as pointed by the author but with no luck (FS just coudn't be mount the second time.
Currently I plan to neglect USE_MAGIC and to replace this code:
if (dat == UINT32_MAX)
{
debugf("First init file system");
spiffs_format_internal(cfg);
writeFirst = true;
}
int res = SPIFFS_mount(&_filesystemStorageHandle,
cfg,
spiffs_work_buf,
spiffs_fds,
sizeof(spiffs_fds),
spiffs_cache,
sizeof(spiffs_cache),
NULL);
debugf("mount res: %d\n", res);
if (writeFirst)
{
file_t fd = SPIFFS_open(&_filesystemStorageHandle, "initialize_fs_header.dat", SPIFFS_CREAT | SPIFFS_TRUNC | SPIFFS_RDWR, 0);
SPIFFS_write(&_filesystemStorageHandle, fd, (u8_t *)"1", 1);
SPIFFS_fremove(&_filesystemStorageHandle, fd);
SPIFFS_close(&_filesystemStorageHandle, fd);
}
with this one:
file_t fd = SPIFFS_open(&_filesystemStorageHandle,"initialize_fs_header.dat",SPIFFS_RDONLY,0);
SPIFFS_close(&_filesystemStorageHandle, fd);
if(fd < 0) {
debugf("No initialized SPIFFS volume found. Initializing FS now...");
file_t fd = SPIFFS_open(&_filesystemStorageHandle, "initialize_fs_header.dat", SPIFFS_CREAT | SPIFFS_TRUNC | SPIFFS_RDWR, 0);
SPIFFS_write(&_filesystemStorageHandle, fd, (u8_t *)"1", 1);
//SPIFFS_fremove(&_filesystemStorageHandle, fd);
SPIFFS_close(&_filesystemStorageHandle, fd);
} else {
debugf("It seems that the mounted SPIFFS is properly inited.");
}
However as indicated in my previous post, if there is some previous data on the flash (wiped flash is all 0xFF as I undestand?) it may lead to unusuable filesystem.
So it is pretty good to erase the flash before use.
@kmihaylov Can you fetch the latest develop
version and check if you still have problems with spiffs?
Hello. Finally I found when my FS is crashing. It is some time after data has been written to it, that's why it was a little bit hard to figure it out when exactly the bug appears.
I found that while there is no need for api_spiffs_erase, the consistency of the FS is OK, the device can be rebooted w/o problems (data loss, broken FS).
On the first appearance of "api_spiffs_erase", however, the FS is immediately broken.
I made a simple program where one can write some data to the FS. Issue command
save 10
where the integer is how many times the fileSetContent will be called. The sample text is in the cpp file and can be changed. I set it to 10 bytes for convenience (idk how SPIFFS reserves bytes for the filename and etc.).You have to try with some random repetition values, 100 is good to begin with. When you figure out where "api_spiffs_erase" is shown you can decrease the argument of
save
to this position -1.I got "api_spiffs_erase" on the 63rd write (630 bytes of data excluding what bytes were reserved for file name, spiffs, etc).
If I try to repeat the file save 62 times, the FS works OK, the device can be safely rebooted w/o data loss. After the reboot only one
save
operation is needed to reach "api_spiffs_erase" and the FS gets immediately broken. On the next reboot it is formatted by the firmware.Example code is provided here: https://github.com/kmihaylov/Sming_fsTest.git
Tested with commit 5300c0b (current develop)