Default support for filesystem (file IO from stdio.h) based tools in ESP-IDF

[nathanjel]

Hi ESP-IDF team,

Based on my current experience ESP-IDF lacks a default filesystem support to allow users utilize libraries that depend on having files to provide data. E.g. mongoose will not allow You to utilize SSL connections out of the box, as it lacks option to load certificates to base supporting library from in-memory string.

While recenty fatfs support was added, it's not provided in a out-of-the-box usable state, as the usage scenario defaults to using external SD card, and if used with default SPI flash it consumes more memory than neccesary to handle files in a pure read-only scenarios, due to immanent filesystem overhead.

I developed a library that addresses this and allows one to build a fixed structure filesystem, which allows to easily include files with directory structure to your application, allows file read/write and also read/write access to SPI flash memory thru familiar fopen/fread/fwrite interface, and ties nicely into current menuconfig and uses ESP VFS for seamless integration.

I have created ESP-IDF compatible component (https://github.com/nathanjel/ffs) to resolve this and using it successfully in my application. I am happy to contribute it to the ESP-IDF project, altought I realize it might take additional effort to make it in-line with the contribution guidelines.

Requesting ESP-IDF team for information, if such extension to the library would be welcomed, and if so I would be happy to polish my library to fully match contribution guidelines and prepare the pull request.

[rojer]

@nathanjel, mongoose-os includes SPIFFS support, mongoose (obviously), structured configuration and OTA on ESP32, ESP8266 and CC3200 out of the box.

[igrr]

@nathanjel Thanks for submitting this library, it has the potential to fill one of the gaps in ESP-IDF functionality 👍

One thing that concerns me a bit at this point is the lack of design doc and tests. Also the 'raw' mapping of files to flash via /raw.dat=r:0x0,g:0x100000 asks to be abused when used in combination with a web server, for example to read out the (possibly encrypted) firmware.

We will take some time to study your code in detail. Due to the upcoming holidays here in China it may take a few weeks before we have some progress on this, though.

[nathanjel]

@igrr I am aware about the documentation gaps. As I use this library I don't need the organized test cases and additional docs. I would be happy to add those. It would be however disappointing to do all this esp-idf specific work and then get the pull request rejected due to it being not aligned with broader concepts of the ESP-IDF library. So I will wait for official "let's do it" :) message prior to spending more time on this.

As for the /raw file, this is a feature of the library, but it is not provided in enabled state in the default package. It would require deliberate actions (create this file, put in the options) to enable it. Furthermore as the raw access is executed thru spiflash* functions that would read out encrypted data. The risk stays however, if one uses the partition based access which then decrypt data on the fly, as in the /ota file usage example. This could be addressed, I hope, by additional option in Kconfig to enable the direct access functionalities, that would display a warning message if checked.

Looking forward for any more feedback.

[rojer]

sorry for the plug here, getting back on topic - SPIFFS works well, has been tested and is pretty easy to integrate - which has been done already, too. its license is permissive, so to me it'd make more sense to have it in the IDF rather than something completely new unless this new thing is compatible with transparent flash encryption. that would be a good reason to design (and test - thoroughly!) a new FS implementation.

[rojer]

ok, i find description confusing. on the one hand, it's "fixed", file set is pre-determined - so no new files? but existing can be rewritten? can one rewrite existing data or is only appending allowed? as already mentioned - how does it interoperate with flash encryption? saying that it uses partition write functions to access flash is not enough, there's more to be encryption-compatible than that.

it needs better user documentation, better technical documentation (description of format), better test suite (it has to exist, for starters). just from cursory examination of sources i can spot a number of feature missing that i would expect a filesystem to have - e.g. O_APPEND support.

[nathanjel]

@rojer, thanks for all the comments

• maximum file size is pre-determined
• no new files, existing can be re-written
• both re-write and appending is possible

O_APPEND could not be fully supported due to limitations of the solution. As the file system is fixed, I only know the initial and maximum file size, thus I could not grant in any way that file position in append mode lands after the last write.

On encryption - I need to test thru the encrypted scenarios yet. Still, looking thru the source code of esp-idf use of esp_partition* functions ensures you get read/write encrypted or unencrypted, depending how partition was defined in the partitioning csv file. You might then configure your files to be on partition which is encrypted or not. Using the default file scenarios or mapping files to parition directly should grant that transparent support of encryption. Mapping files to memory gives You raw memory data, with no encryption support. Since You mention there is more needed than just that to be encryption-compatible, please be so kind to share the details.

I am also very grateful for the time You took to review the source. Would You be so kind to create an issue directly in my project listing the other items You spotted? It will be of great help in continued work on the library.

Also, I have nothing against seeing SPIFFS as the default solution in ESP-IDF, and even for my project I was first considering to copy over a port from https://github.com/whitecatboard/Lua-RTOS-ESP32/tree/master/components/spiffs which seems to work and do the job. But my concerns were also on SPI flash memory usage, I wanted to make sure i get minimum needed space occupied by my additional files, as I use a significant portion of SPI flash memory to store metering data. SPIFFS, as any other FS would, has a filesystem block and uses pages/blocks so by default brings overhead, while my solution allowed me all the "static" files to be kept without any overhead.

[rojer]

appending is an important use case. think log file - i want to fopen("my.log", "a") and just start writing.

encryption breaks NOR flash semantics and turns byte-wise write access into block-wise, in case of esp flash encryption - into 32-byte sized, 32-byte aligned access. there are implications for that when writing is concerned. see discussion here as it applies to spiffs.

as far as overhead is concerned - i pulled quick stats on my FS image:

Image stats: size=131072, space: total=113201, used=12048, free=101153

so 86% of the space allocated to fs is available to the user. that's a-ok for most users, myself included. if you need that extra few percent - sure. but i wouldn't trade that for not being able to create new files and limiting existing file sizes upfront. that narrows usage pretty severely.

actually, what might make sense as a default filesystem fro esp-idf is "NVS fs" - VFS interface to NVS. that should be simple to do, NVS is already there and will soon support encryption. with that, it may be all we need. once NVS encryption support lands, i'll probably take a stab at it.

[nathanjel]

While not the perfect solution, I do manipulate on 4k flash pages, doing erase and re-write of the whole page. I use it for configuration files, where i update the whole configuration writing the new one from file to memory. No 32-byte issue here.

The overhead topic is more complicated than just the initial overhead of the FS file allocation table. As FS usually operates on blocks, each file added to it uses up it's own space up to a block boundary. In a web server scenario, there are usually multiple small files, so the overhead goes up.

The usage scenarios which You bring up justify bringing in a fully functional file system.

[nathanjel]

Had a look at the NVS, it would seem quite complicated. E.g. if You map one file = one nvs entry, you would have to use the blob type and each time to change a file, it would take a complete rewrite of it. It also does not provide the entry listing by default.

[igrr]

After internal discussion we have decided to introduce a flash translation layer so that we can have FATFS on top of (possibly encrypted) SPI flash. Corresponding changes are on their way into master now.

Closing this issue, but it would be awesome if you could add the component to the list of third party components we are assembling here: https://esp32.com/viewtopic.php?f=2&t=1498

[rojer]

flash translation layer? sounds interesting. i presume it will take care of wear-leveling and such?

[igrr]

It takes care of wear levelling, plus some (limited) consistency checks.

[rojer]

cool, thanks

[Ritesh236]

Hi,

We have started to work on SPIFFS component which is provided into ESP32 IDF 3.0 but we found below statement into its release note

Presently, spiffs does not support directories.

So, We need that support for our project requirement in which we need to use SPIFFS as file system to store some files and certificates as per project requirement

Does anyone has implemented and modified it? If yes then please provide link or files for that

Let me know if need any information from my side for that