SPIFFS problems

[boneskull]

I can use mkspiffs to build the SPIFFS filesystem image. esptool can flash it. In other words, make flashdata is successful, AFAICT.

However, upon mount, SPIFFS doesn't recognize the filesystem; it raises SPIFFS_ERR_NOT_A_FS:

D (354) duktape_spiffs: Loading SPIFFS at address 0x180000
E (364) duktape_spiffs: Error: rc from spiffs mount = -10025

I'm wondering if this could be a mismatch between the mkspiffs version and the SPIFFS version. My mkspiffs is v0.2.0; I have no idea what version is used in components/spiffs/. I know there are some backwards-compat concerns with SPIFFS.

I may be able to work around this by modifying duktape_spiffs.c to reformat, but that would necessarily remove all of the files in the image, and I'd have to upload them again manually (I think). Otherwise, I'll try to update the spiffs component, in case it's old.

[boneskull]

OK, so the spiffs component isn't from ESP-IDF's implementation, and I'm not sure what version of mkspiffs is in bin/ without running it in a VM (bin/mkspiffs --version). I could probably snag that version, then, and recompile it for MacOS to see if it'll make a reasonable FS.

Not sure if ESP-IDF's implementation can be used, but if it can, duktape_spiffs.c needs refactors to consume it. Or, if the Lua RTOS SPIFFS implementation has an update, maybe that's a better idea? But I can't be sure this is the problem I'm seeing; it's just a hunch.

[boneskull]

Insofar as some of the tooling/frameworks are concerned, maybe it'd be possible to consume PlatformIO to manage it, as to avoid committing blobs to VCS... it supports ESP-IDF, but I have not yet tried to use that framework with it.

[nkolban]

The usage of the spiffs in this project pre-dates the availability of the ESP-IDF SPIFFs. An alternate version from elsewhere was used. What we probably want to do is replace any usage of spiffs processing in this project with the now official / formal ESP-IDF SPIFFS functions found in ESP-IDF.

[boneskull]

@nkolban think we can do away with actually needing spiffs.h itself? that would certainly simplify things.

[boneskull]

OK, my PR #25 seems to have broken reading. I'll look into this.

also I'll disable formatting upon error. even though duktape-esp32 won't be able to read the filesystem, it should be salvageable thru other means.

[dashxdr]

When my partitions line had a size 0x180000: SPIFFS,data,SPIFFS,0x180000,0x180000, # WRONG I got the same SPIFFS_ERR_NOT_A_FS you mentioned earlier. Setting the partition size to 0x80000 to match the mkspiffs command fixed that issue. SPIFFS,data,SPIFFS,0x180000,0x80000, # WORKING...

[boneskull]

wtf... the partitions.csv is a 0-byte file. oops.

[boneskull]

that's what I had in there, but I botched the commit.

[boneskull]

this is my partitions.csv. but reading a file results in an EIO "I/O Error". I'm unfamiliar with debugging these things, but I'll give it a shot later tonight or tomorrow if I can find some time.

nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 1M,
spiffs,   data, spiffs,  0x180000, 0x80000

it appears it stats the file fine:

D (25743) module_fs: << js_fs_statSync: /spiffs/web/ide.html - size: 6018

this is the correct size. js_fs_openSync returns a file handle, but then...

D (25873) module_fs: Buffer pointer size returned as 128
D (25873) module_fs: js_fs_readSync: read() error: 5 I/O error
D (25883) module_fs: << js_fs_readSync: sizeRead: 0

[dashxdr]

I think the problem is the spiffs.img generated by mkspiffs is not consistent with the esp-idf spiffs configuration or implementation. I've tried various mkspiffs versions and the tool is intrinsicly flaky. For example no version will succeed in doing a -c of a directory tree into an spiffs.img and then -u that into a directory... which is a basic check for sanity. Meaning if the tool itself is unable to parse its own generated image, what hope will esp-idf have?

[boneskull]

I thought SPIFFS was flat?

[dashxdr]

I remember seeing such an assertion somewhere... yeah, here it is: https://github.com/pellepl/spiffs/tree/f5e26c4e933189593a71c6b82cda381a7b21e41c#features That claims tmp/myfile.txt will become myfile.txt BUT that isn't consistent with observations. The stat of web/ide.html succeeds, but the read of it fails. I even tried accessing files in the top level (no luck). And accessing that file as ide.html (no luck).

The failure occurs in esp-idf/components/spiffs/spiffs/src/spiffs_nucleus.c the function spiffs_page_data_check returns SPIFFS_ERR_INDEX_REF_LU.

[dashxdr]

OK I think the problem is the mkspiffs is just not the matching tool for the espressif/esp-idf source. The fact that we get any life at all is pure luck, it seems. The default behaviour of mounting the spiffs is to format on failure. Then you get a writable flash file system, and you can manually populate its contents.

However https://github.com/pellepl/spiffs/#building mentions spiffsimg https://github.com/nodemcu/nodemcu-firmware/tree/master/tools/spiffsimg This is the equivalent of mkspiffs but for the built in esp-idf spiffs component. Somehow this tool would need to be integrated somewhere. I'd suggest duktape-esp32 itself.

My application for the esp32 doesn't even require writing to the flash, or the web ide. I just need bluetooth, so I'm moving onto other tasks. Good luck!

[dashxdr]

Ok a thought occured to me. Since the spiffs is both read/write and it is designed to be embedded, the purpose of the spiffsimg tool is to just embed spiffs writing into a 'c' tool on the workstation itself (not the target). So building the spiffsimg tool would be just using the esp-idf's copy of spiffs source files with the right glue.

I am trying to devise a development setup where I can start to flesh out a javascript application built on top of the duktape-esp32 infrastructure. I can add 'c' glue code as necessary but the bulk needs to be in javascript (I need to pass this off to people who know javascript well but 'c' not so much...). The serialport http machinery to run a single javascript file is useful for simple tests, but not for real development.

So might there not be a common solution? It occured to me to leverage the WIFI support. If I had the spiffsimg tool working on my desktop I could have a small program that would deliver a spiffs image file over the network (spiffs network server). On the ESP32 it would have a spiffs network client that talks to the server, and this network spiffs filesystem could be mounted somewhere (like /net). A config option in NVS could control this (server IP:PORT).

This would avoid all wear and tear on the flash device, would be faster than flash writes, and to test out a new version would just require reboot of the ESP32. Once the filesystem tree is stable the spiffs.img could be written to the ESP32 and NVS config could be switched back to use that over wifi/net

[boneskull]

Here's what I'm not understanding: why do we save the files to both places? @nkolban explains in the docs why there are two filesystems:

Unfortunately, we can't just use ESPFS as our file system in place of SPIFFS because ESPFS is read-only. We can built our file system image of files at Duktape build time but can't dynamically write to it.

It would seem to me that we should favor ESPFS for the "built in" files. I'm not sure why we put the files in both places. What if, when we want a file via the web server, it first looks in SPIFFS, and if that fails, it tries to load the file from ESPFS? Then we wouldn't need to duplicate the files, and users can override behavior. Dunno--just an idea.

I can't presently verify that my code actually works, assuming the filesystem has been formatted properly, because I'm unsure how to push files to SPIFFS otherwise. May be able to leverage the websocket server?

I like @dashxdr's idea too.

@dashxdr Are you sure it's the image that's the problem, and not my code? (assuming a correct partition setup)

[boneskull]

@dashxdr I fixed it. PR incoming.

[dashxdr]

NK goes into details here: https://github.com/nkolban/duktape-esp32/blob/master/docs/file%20systems.md and I didn't notice it before but he included a link to the spiffs source (which has mkspiffs in it) https://github.com/whitecatboard/Lua-RTOS-ESP32/tree/master/components

[dashxdr]

Still not working for me. I did get the most recent esp-idf and made your sdkconfig tweaks CONFIG_SPIFFS_META_LENGTH=0 behaviour is the same though. D (30786) SPIFFS: read: offset:0 rd:128 data spix:0000 is data_pix:0000 addr:00000005

esp-idf commit hash a5eb369126e5cef31715d2755931cf933bb98bee xtensa gcc 1.22.0-80-g6c4433a my duktape-esp32 fork here: https://github.com/dashxdr/duktape-esp32 just...doesn't...work...

[dashxdr]

Ok found it. https://github.com/igrr/mkspiffs/releases/tag/0.2.1 mkspiffs 0.2.1 required Just got the PR going...

[boneskull]

this can be closed for now I believe.