search

nodemcu / nodemcu-firmware

Lua based interactive firmware for ESP8266, ESP8285 and ESP32
https://nodemcu.readthedocs.io
MIT License
7.67k stars 3.13k forks source link

Filesystem won't cooperate when flash is maxed out #246

Closed raz123 closed 9 years ago

raz123 commented 9 years ago

Looks like I'm unable to interact with the filesystem at a certain point. I'll see if I can reproduce. In the meantime, here's what's happening:

NodeMCU 0.9.5 build 20150213  powered by Lua 5.1.4
> for k,v in pairs(file.list()) do l = string.format("%-15s",k) print(l.."   "..v.." bytes") end

cwide.lc          2526 bytes
cwide.lua         5546 bytes
ckb5p2.lua        1389 bytes
translate.lua     496 bytes
kb.txt            79647 bytes
LLbin.lua         639 bytes

> file.remove("cwide.lc")
> for k,v in pairs(file.list()) do l = string.format("%-15s",k) print(l.."   "..v.." bytes") end

cwide.lc          2526 bytes
cwide.lua         5546 bytes
ckb5p2.lua        1389 bytes
translate.lua     496 bytes
kb.txt            79647 bytes
LLbin.lua         639 bytes

As you can see, I am unable to remove files. On the other hand, I am able to rename files without a problem. I suspect that this might be because I have maxed out the flash.

I also tried issuing this command, which usually resets a file to 0 bytes. It did not work.

file.open('(filename)', 'w')
file.close()

Finally, I had to issue file.format() to get it working again.

raz123 commented 9 years ago

Had it happen again, after the format. This time, the flash is far from being full.

Log:

NodeMCU 0.9.5 build 20150213  powered by Lua 5.1.4
lua: cannot open init.lua
> for k,v in pairs(file.list()) do l = string.format("%-15s",k) print(l.."   "..v.." bytes") end

kb.txt            6350 bytes
LLbin.lua         639 bytes
ckb5p2.lua        1391 bytes

> file.rename("kb.txt", "kb2.txt")
> for k,v in pairs(file.list()) do l = string.format("%-15s",k) print(l.."   "..v.." bytes") end

ckb5p2.lua        1391 bytes
LLbin.lua         639 bytes
kb2.txt           6350 bytes

> file.remove("kb2.txt")
> for k,v in pairs(file.list()) do l = string.format("%-15s",k) print(l.."   "..v.." bytes") end

ckb5p2.lua        1391 bytes
LLbin.lua         639 bytes
kb2.txt           6350 bytes
marcoskirsch commented 9 years ago

I've seen this as well. I just format and start over. But I can see this being a huge problem to someone writing files in their code.

mikewen commented 9 years ago

It would be nice if there is a api to check free space on flash.

pellepl commented 9 years ago

Hi, this is odd. If you ever get this again, pretty please take a dump of the spiffs region of the flash and bring it to me as a bugreport in the spiffs repo. Also, if you find a way to reproduce it, please file a bug describing how you did it. I would also need the configuration. Thanx!

xlfe commented 9 years ago

Anyone know how to take a dump of the spiffs region? I couldn't figure it out

pellepl commented 9 years ago

Well.. I am a total noob to this platform but looking at the spiffs integration file app/spiffs.c it seems one could do, in c:

platform_flash_read(dst, fs.cfg.phys_addr, fs.cfg.phys_size);

where dst would be some memory. How to bind it to lua I have no idea.

Also, I guess you'd get a pretty big area, so you'd probably need to do this in chunks rather than in one big blob.

How to get it to the pc on the ESP platform - this is a wifi module right, perhaps you could ftp it? Otherwise, just printing out the hexvalues would also do, I could write a script to hex2bin it.

Just a thought ;)

marcoskirsch commented 9 years ago

Connect GPIO0 to GND and use esptool.py.

raz123 commented 9 years ago

Connect GPIO0 to GND and use esptool.py.

Any clue as to the approximate address/size NodeMcu starts its SPIFFS fs? Source code seems to have it look for a free sector before establishing the zone.

xlfe commented 9 years ago

@marcoskirsch ah - hmm for some reason I thought you could only write with esptool.py - glad to be proven wrong

@raz123 you can calculate the offset thus:

#size of 0x10000.bin
sz = 285312
adj = sz/0x4000*0x4000+0x4000
print 'Write to {:02x}'.format(adj+0x10000)
raz123 commented 9 years ago

So I've been trying to reproduce FS errors and I came up with this one. I don't have flash dumps for the moment, but hopefully Nodemcu devs can look into this:

Obvservations:

NodeMCU 0.9.5 build 20150213  powered by Lua 5.1.4
lua: cannot open init.lua

> for n,s in pairs(file.list()) do print(n.." size: "..s) end
# List shows no files present on FS.

> file.open("test1.txt", "a+") for i = 1, 100*1000 do file.write("x") end file.close() print("Done.")
Done.
# We purposefully write more than the FS can handle (100k bytes).

> for n,s in pairs(file.list()) do print(n.." size: "..s) end
test1.txt size: 66048
# List shows that the file only wrote 66048 bytes, which seems like the available space on the FS.

> file.remove("test1.txt")
# We remove the file we just created.

> for n,s in pairs(file.list()) do print(n.." size: "..s) end
# List shows no files present on FS.

> file.open("test2.txt", "a+") for i = 1, 1*1000 do file.write("x") end file.close() print("Done.")
stdin:1: open a file first
# We try to write a 1k bytes file and we are greeted with an error.

Restarting the module will not help either.

Conclusion: The FS cannot be further written to unless a file.format() command is issued.
pellepl commented 9 years ago

Great, I'll make a testcase out of it and see if I can reproduce it. I'll let you know of any progress. Thanks! Den 7 mar 2015 02:22 skrev "raz123" notifications@github.com:

So I've been trying to reproduce FS errors and I came up with this one. I don't have flash dumps for the moment, but hopefully Nodemcu devs can look into this:

Obvservations:

NodeMCU 0.9.5 build 20150213 powered by Lua 5.1.4 lua: cannot open init.lua

for n,s in pairs(file.list()) do print(n.." size: "..s) end

List shows no files present on FS.

file.open("test1.txt", "a+") for i = 1, 100*1000 do file.write("x") end file.close() print("Done.") Done.

We purposefully write more than the FS can handle (100k bytes).

for n,s in pairs(file.list()) do print(n.." size: "..s) end test1.txt size: 66048

List shows that the file only wrote 66048 bytes, which seems like the available space on the FS.

file.remove("test1.txt")

We remove the file we just created.

for n,s in pairs(file.list()) do print(n.." size: "..s) end

List shows no files present on FS.

file.open("test2.txt", "a+") for i = 1, 1*1000 do file.write("x") end file.close() print("Done.") stdin:1: open a file first

We try to write a 1k bytes file and we are greeted with an error.

Restarting the module will not help either.

Conclusion: The FS cannot be written to unless a file.format() command is issued.

— Reply to this email directly or view it on GitHub https://github.com/nodemcu/nodemcu-firmware/issues/246#issuecomment-77665163 .

pellepl commented 9 years ago

Hi,

I've pushed some changes. This would do it I think. Please try it out.

Cheers / Peter

2015-03-08 9:00 GMT+01:00 Peter Andersson pelleplutt1976@gmail.com:

Great, I'll make a testcase out of it and see if I can reproduce it. I'll let you know of any progress. Thanks! Den 7 mar 2015 02:22 skrev "raz123" notifications@github.com:

So I've been trying to reproduce FS errors and I came up with this one. I

don't have flash dumps for the moment, but hopefully Nodemcu devs can look into this:

Obvservations:

NodeMCU 0.9.5 build 20150213 powered by Lua 5.1.4 lua: cannot open init.lua

for n,s in pairs(file.list()) do print(n.." size: "..s) end

List shows no files present on FS.

file.open("test1.txt", "a+") for i = 1, 100*1000 do file.write("x") end file.close() print("Done.") Done.

We purposefully write more than the FS can handle (100k bytes).

for n,s in pairs(file.list()) do print(n.." size: "..s) end test1.txt size: 66048

List shows that the file only wrote 66048 bytes, which seems like the available space on the FS.

file.remove("test1.txt")

We remove the file we just created.

for n,s in pairs(file.list()) do print(n.." size: "..s) end

List shows no files present on FS.

file.open("test2.txt", "a+") for i = 1, 1*1000 do file.write("x") end file.close() print("Done.") stdin:1: open a file first

We try to write a 1k bytes file and we are greeted with an error.

Restarting the module will not help either.

Conclusion: The FS cannot be written to unless a file.format() command is issued.

— Reply to this email directly or view it on GitHub https://github.com/nodemcu/nodemcu-firmware/issues/246#issuecomment-77665163 .

nodemcu commented 9 years ago

@pellepl thanks very much. I have build a firmware for every one to download and test. download

raz123 commented 9 years ago

@pellepl thanks, seems to have fixed that one case. I stumbled upon another one:

NodeMCU 0.9.5 build 20150310  powered by Lua 5.1.4
lua: cannot open init.lua

> for n,s in pairs(file.list()) do print(n.." size: "..s) end
# List: empty FS.

> file.open("test1.txt", "a+") for i = 1, 70*1000 do file.write("x") end file.close() print("Done.")
Done.
# We write a 70k bytes file.

> for n,s in pairs(file.list()) do print(n.." size: "..s) end
test1.txt size: 59136
# List: only a ~59k bytes file made it.

> file.open("test2.txt", "a+") for i = 1, 1*1000 do file.write("x") end file.close() print("Done.")
Done.
# We (try to) write a 1k bytes file.

> for n,s in pairs(file.list()) do print(n.." size: "..s) end
test1.txt size: 59136
test2.txt size: 0
# List: new file was created, but is empty.

> file.remove("test1.txt")
> file.remove("test2.txt")
# We (try to) remove the two files.

> for n,s in pairs(file.list()) do print(n.." size: "..s) end
test1.txt size: 59136
test2.txt size: 0
# List: the two files are still present.

At this point, nothing will remove the two files besides a file.format().

pellepl commented 9 years ago

Thanks for the confirmation, and good bug hunting - darn! :)

I'm on to it, I have a hunch of what it may be.

If possible, could anyone give me the contents of the spiffs struct? Or in particular: spiffs.cfg.phys_size spiffs.cfg.phys_erase_block spiffs.cfg.log_block_size spiffs.cfg.log_page_size spiffs.block_count

My efforts of being bit-exact fails pretty bad right now. Though it seems I can reproduce the bugs anyway, but still..

Cheers / P

2015-03-09 19:12 GMT+01:00 raz123 notifications@github.com:

@pellepl https://github.com/pellepl thanks, seems to have fixed that one case. I stumbled upon another one:

NodeMCU 0.9.5 build 20150310 powered by Lua 5.1.4 lua: cannot open init.lua

for n,s in pairs(file.list()) do print(n.." size: "..s) end

List: empty FS.

file.open("test1.txt", "a+") for i = 1, 70*1000 do file.write("x") end file.close() print("Done.") Done.

We write a 70k bytes file.

for n,s in pairs(file.list()) do print(n.." size: "..s) end test1.txt size: 59136

List: only a ~59k bytes file made it.

file.open("test2.txt", "a+") for i = 1, 1*1000 do file.write("x") end file.close() print("Done.") Done.

We (try to) write a 1k bytes file.

for n,s in pairs(file.list()) do print(n.." size: "..s) end test1.txt size: 59136 test2.txt size: 0

List: new file was created, but is empty.

file.remove("test1.txt") file.remove("test2.txt")

We (try to) remove the two files.

for n,s in pairs(file.list()) do print(n.." size: "..s) end test1.txt size: 59136 test2.txt size: 0

List: the two files are still present.

At this point, nothing will remove the two files besides a file.format().

— Reply to this email directly or view it on GitHub https://github.com/nodemcu/nodemcu-firmware/issues/246#issuecomment-77909931 .

nodemcu commented 9 years ago

hi, here is the config.

#define LOG_PAGE_SIZE       256
static u8_t spiffs_work_buf[LOG_PAGE_SIZE*2];
static u8_t spiffs_fds[32*4];
static u8_t spiffs_cache[(LOG_PAGE_SIZE+32)*4];
...
#define SPI_FLASH_SEC_SIZE      4096
...
#else
#define FLASH_SEC_NUM   0x80
#endif
#define SYS_PARAM_SEC_NUM 4
#define SYS_PARAM_SEC_START (FLASH_SEC_NUM - SYS_PARAM_SEC_NUM)
// SYS_PARAM_SEC_START is at the end of flash.
#define INTERNAL_FLASH_SIZE             ( (SYS_PARAM_SEC_START) * INTERNAL_FLASH_SECTOR_SIZE )
#define INTERNAL_FLASH_START_ADDRESS    0x40200000
#define INTERNAL_FLASH_SECTOR_SIZE      SPI_FLASH_SEC_SIZE
...
  spiffs_config cfg;
  cfg.phys_addr = ( u32_t )platform_flash_get_first_free_block_address( NULL ); 
  cfg.phys_addr += 0x3000;
  cfg.phys_addr &= 0xFFFFC000;  // align to 4 sector.
  cfg.phys_size = INTERNAL_FLASH_SIZE - ( ( u32_t )cfg.phys_addr - INTERNAL_FLASH_START_ADDRESS );
  cfg.phys_erase_block = INTERNAL_FLASH_SECTOR_SIZE; // according to datasheet
  cfg.log_block_size = INTERNAL_FLASH_SECTOR_SIZE; // let us not complicate things
  cfg.log_page_size = LOG_PAGE_SIZE; // as we said
  NODE_DBG("fs.start:%x,max:%x\n",cfg.phys_addr,cfg.phys_size);

the output will be: fs.start: 0x40268000, max: 0x14000

there is also some diffs:

#define SPIFFS_CACHE_STATS              0
#define SPIFFS_GC_STATS                 0
//s32_t SPIFFS_read(spiffs *fs, spiffs_file fh, void *buf, s32_t len) to
s32_t SPIFFS_read(spiffs *fs, spiffs_file fh, void *buf, u32_t len)

//s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) to
s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, u32_t len)
pellepl commented 9 years ago

Many thanks!

Upped spiffs again, this should take care of most/all/none of full filesystem flukes.

@raz123: It can be possible to fill the system with one file, and then to create another one.

To append to a file, spiffs check for two free pages (one for data, one for updating the file header page). When creating a new dataless file, only one page is needed, hence it is possible to create yet another file after getting SPIFFS_ERR_FULL, though it may be confusing. However, adding data to this new file won't work.

Hope it works now, and thank you guys again for all help! If you find more stuff, just shout.

Cheers / Peter

2015-03-10 2:14 GMT+01:00 zeroday notifications@github.com:

hi, here is the config.

define LOG_PAGE_SIZE 256

static u8_t spiffs_work_buf[LOG_PAGE_SIZE_2]; static u8_t spiffs_fds[32_4]; static u8_t spiffs_cache[(LOG_PAGE_SIZE+32)*4]; ...

define SPI_FLASH_SEC_SIZE 4096

...

else

define FLASH_SEC_NUM 0x80

endif

define SYS_PARAM_SEC_NUM 4

define SYS_PARAM_SEC_START (FLASH_SEC_NUM - SYS_PARAM_SEC_NUM)

// SYS_PARAM_SEC_START is at the end of flash.

define INTERNAL_FLASH_SIZE ( (SYS_PARAM_SEC_START) * INTERNAL_FLASH_SECTOR_SIZE )

define INTERNAL_FLASH_START_ADDRESS 0x40200000

define INTERNAL_FLASH_SECTOR_SIZE SPI_FLASH_SEC_SIZE

... spiffs_config cfg; cfg.phys_addr = ( u32_t )platform_flash_get_first_free_block_address( NULL ); cfg.phys_addr += 0x3000; cfg.phys_addr &= 0xFFFFC000; // align to 4 sector. cfg.phys_size = INTERNAL_FLASH_SIZE - ( ( u32_t )cfg.phys_addr - INTERNAL_FLASH_START_ADDRESS ); cfg.phys_erase_block = INTERNAL_FLASH_SECTOR_SIZE; // according to datasheet cfg.log_block_size = INTERNAL_FLASH_SECTOR_SIZE; // let us not complicate things cfg.log_page_size = LOG_PAGE_SIZE; // as we said NODE_DBG("fs.start:%x,max:%x\n",cfg.phys_addr,cfg.phys_size);

the output will be: fs.start: 0x40268000, max: 0x14000

there is also some diffs:

define SPIFFS_CACHE_STATS 0

define SPIFFS_GC_STATS 0

//s32_t SPIFFS_read(spiffs fs, spiffs_file fh, void buf, s32_t len) to s32_t SPIFFS_read(spiffs fs, spiffs_file fh, void buf, u32_t len)

//s32_t SPIFFS_write(spiffs fs, spiffs_file fh, void buf, s32_t len) to s32_t SPIFFS_write(spiffs fs, spiffs_file fh, void buf, u32_t len)

— Reply to this email directly or view it on GitHub https://github.com/nodemcu/nodemcu-firmware/issues/246#issuecomment-77978289 .

nodemcu commented 9 years ago

newest fix update, pre-built links: download it works great in my little test. Thanks @pellepl

raz123 commented 9 years ago

@marcoskirsch, would you mind telling us the params that you use with esptool.py to dump the flash area? My futile attempts only managed to fetch a bunch of zeroes. Thanks!