overlay filesystem

[phatpaul]

I use espfs as the base read-only filesystem which holds factory-default files for my project. Then I use a FAT fs at run-time to store any changes. When a file is accessed, I have logic to first check FAT to see if the file exists there, then if not try to read it from espfs. So it is essentially an OverlayFS. This enables me to perform a factory-reset (clear all user settings) by just formatting the FAT partition and rebooting.

To avoid having to duplicate this logic (httpd and other tasks), I'm thinking it might be appropriate to add it to the espfs_vfs layer in this module.
So you would initialize espfs_vfs with an optional parameter that specifies the upper/overlay filesystem path, like this:

// Mount trivial read-only filesystem: espfs
esp_vfs_espfs_conf_t espfs_vfs_conf = {
  .base_path = "/espfs",
  .max_files = 5,
  .overlay_path = "/fatfs"
};
esp_vfs_espfs_register(&espfs_vfs_conf);

vfs_espfs_open() will check if overlay_path is not NULL, then somehow forward the open call to the upper/overlay filesystem. If the file is not found in upper, then handle as usual via espfs.

I'm sure I haven't thought this completely through... Perhaps it should be a separate OverlayFS module that specifies both "upper" and "lower" fs? Maybe it already exists? Thoughts?

[jkent]

Hmm, I like the idea. My first reaction was put it in a different module, but I then thought about the lack of utility gained with it in a separate submodule. The only reason I think to have it in a submodule is if more than 2 layers would be desired.

[jkent]

I will be addressing this with espfs v2. I forgot I started working on it. @phatpaul, could you take a look at my progress with libesphttpd?

https://github.com/jkent/esphttpd-example https://jkent.net/~jkent/libesphttpd/

[phatpaul]

Wow, you've been busy. I'm pretty busy trying to finish a new version of my product, still using IDF 3.3.3.
Hopefully when the dust settles here I'll try to upgrade to IDF 4 again.

Can you please summarize what all you've changed recently in both espfs and libesphttpd? (or is that already documented somewhere?)

[jkent]

Well for libespfs, I've got documentation here https://libespfs.readthedocs.io/en/stable/ For libesphttpd, https://jkent.net/~jkent/libesphttpd/ -- we'll get it on readthedocs.io after the rewrite is complete

libesphttpd is still a work in progress, there are a few issues in the rewrite I'd still like to address. It uses mbedtls directly now, and handles the connection pool MUCH nicer. For example, if there is no connections available, but one of the connections in the pool is not in the middle of a request it is killed off and recycled.

[jkent]

@phatpaul if you would look this over that would be fantastic. I think it meets your requirements.

[phatpaul]

Sorry @jkent I'm swamped at the moment. And I don't see that changing any time soon (seems there's a global chip shortage so I have to port a bunch of products to some other chips that my company happened to get a hold of).

Is there anyone else who can help validate it?

[jkent]

Don't worry about it! Its all good. Maybe @chmorgan can look at it.

[jkent]

I've put together a Vue.js application to test and exercise the vfs code. It just needs backend code to be written. This is a priority after making directory entries an optional feature.