use with libesphttpd

[phatpaul]

I'm interested in accessing the espfs via vfs.

My project uses Espfs built-in to libesphttpd https://github.com/chmorgan/libesphttpd but I want to access it using VFS from other modules in my project.

Can this module coexist with libesphttpd ? Would it make sense to modify libesphttpd to use a external git module like this (instead of including the espfs code)?
@chmorgan thoughts?

[chmorgan]

@phatpaul which external git module are you proposing to use? Or are you saying you are using libesphttpd/espfs in your project and you are wondering if that folder should be broken out into its own repository?

[jkent]

@phatpaul this repo is designed to be a IDF component, not a bare submodule. Might be able to make it serve both roles though. I haven't tested it with chmorgan's libesphttpd fork, I think it would work though.

[chmorgan]

Why don’t we ask the user to use this component rather than including it as a sub module in libesphttpd? That’s what other components have done, when there are separate repositories available.

[phatpaul]

@chmorgan I think I understand what you're saying. So we would remove the ESPFS stuff from libesphttpd repo. The user's project (i.e. the example project esphttpd-freertos) would include a espfs submodule which libesphttpd could use. But we would need to resolve the dependency of libesphttpd on espfs in the case that the user didn't include espfs module.

Like this? /main/ <-- user's code /components/libesphttpd/ <-- git submodule /components/espfs/ <-- git submodule

[chmorgan]

Yep, like David antliffs one wire ds18b20 component requires his other lower level component.

https://github.com/DavidAntliff/esp32-ds18b20/blob/master/README.md

[chmorgan]

We would mention needing the other component in the readme. We should be able to add a dependency on that module in the components file right? Like any other required component?

[phatpaul]

OK thanks. I will play around with this arrangement since I'm needing to share espfs between httpd and another (bluetooth) module in my project. And I want to access espfs via VFS in my bluetooth module.

I assume we want to continue the raw espfs api access (not via VFS) in libesphttpd. So those need to be exposed in the submodule.

I don't want to break ESP8266 support, but I don't have any experience with that platform. What should I watch out for?

[chmorgan]

@phatpaul what would be the tradeoff between raw vs. vfs?

On ESP8266 support I think there is most development effort around esp32 these days, I think we would want help from esp8266 users to fix that target. It would be nice not to break it but we shouldn't hold up development on esp32 features to avoid that.

[phatpaul]

It looks like VFS just adds a layer of abstraction. It adds the code here: https://github.com/jkent/esp32-espfs/blob/master/esp_espfs.c - some indirection and an info struct in RAM.

I don't think it would benefit libesphttpd to remove the raw espfs access, except maybe cleaning-up/consolidating the code.

Anyway libesphttpd already has a VFS cgi example (in addition to espfs), so it's up to the user to choose how to access espfs (directly or via vfs).

[phatpaul]

So turns out there is a good reason to keep the raw espfs access for httpd. It provides the gzip flag which tells httpd to send a gzip header to the client.

I just tried using a VFS to espfs connector for httpd and I'm just seeing garbled text instead of index.html. I think it's because VFS doesn't know that index.html has been gzip compressed, so no header is sent via httpd. https://github.com/chmorgan/libesphttpd/blob/master/core/httpdespfs.c#L135

So if using VFS then logic in httpd is looking for a file with .gz extension. https://github.com/chmorgan/libesphttpd/blob/master/util/esp32_httpd_vfs.c#L90

I guess mkespfsimage should have an option to append .gz to filename??

[jkent]

I checked for the gzip magic bytes in my fork of libesphttpd: https://github.com/jkent/esp32-esphttpd/blob/master/util/esp32_httpd_vfs.c#L93

Perhaps this should be done along with a file extension check? I think urls with .gz are kinda ugly.

[chmorgan]

@jkent, @phatpaul do Nginx and Apache use the magic bytes? That could be helpful info.

Imo we could easily use magic bytes. @jkent is that something you could create a PR for to merge back into the main repo?

[jkent]

@chmorgan with mime magic I think apache does, but it probably checks if its text or binary first.

Yeah, I think extension + magic bytes would the best way to go. mkespfsimage only gzip compresses if the file extension is in "html,css,js,svg" by default. I'll work on this now.