support ESP-IDF 5.x

[phatpaul]

I'm finally upgrading my project to ESP-IDF 5.x and it looks like this library will need some changes to support it.

The first issue I encounter is

Failed to resolve component 'openssl'.

Looking at the 4.4 to 5.0 migration guide:

OpenSSL-API component is no longer supported. It is not available in the IDF Component Registry, either. Please use ESP-TLS or mbedtls API directly.

I see that @jkent has already worked on removing openssl on his fork: https://github.com/jkent/cwhttpd/commit/4da0313d73d95e21bef89988b8a1495158f3cb13 (I wish he had contributed his fixes back to this project.)

I assume that's not the only issue that I will have. I'll document them here as I go...

[jkent]

Hi @phatpaul

I realize now my fork was hostile. I felt things were not moving fast enough. I should have been less impatient. I also didn't realize I had the ability to create branches here.

I recently started work on frogfs (libespfs) again -- latest work is in the 'next' branch. I had plans to revisit the libesphttpd situation after I finish the vfs overlay test code

[chmorgan]

@jkent @phatpaul in terms of maintainer ship, you both have been doing a lot of work here and keeping things going. Interestingly I'm back working on esp32 in my daily job so who knows if I'll have need to use this library in the future.

I think you are both maintainers already (just confirmed and you are) and I'd gladly transfer the project out from my personal account and over to a libesphttpd group to reflect that its not just my personal project.

Also, any thoughts on using parts of httpd from esp-idf in the library? I wasn't surprised, but it seemed like a bit of a waste, that they'd just go and develop their own implementation without collaborating here but...

[jkent]

Hey @chmorgan

I'm okay with that. I'm not sure on the naming though. 'libesphttpd' doesn't really roll off the tongue. I had to write it down for my nerd cousin that is hard of hearing to know what I was saying. Like frogfs, I was working to make the code not require IDF, but also work stand-alone.

As for the httpd included in IDF, it has a completely different design, I'm not sure how useful it would be. I think its a bit harder to develop for as well.

[chmorgan]

@jkent a rename sounds fine to me, naming ideas? @phatpaul?

I think the various plugins that libesphttpd has could be a good reason people use the project but I haven't looked at downloads/stars recently, like ability to configure wifi etc.

and was throwing out httpd in case there might be overlap at this point, like when c++ added support for mutexes and you can drop using pthread calls directly. I haven't used httpd myself and only briefly looked at the api a few years back.

[phatpaul]

Glad to hear you are active again @jkent I like the progress that you have made on the project and I'm ok with the name change, as long as we agree to go in the same direction.

@chmorgan I like the idea of transferring the httpd project to a group account. Do you want to do that and then we can decide on how to move the project forward? My vote is to adopt all of jkent's changes and keep the name change.

The ESP-IDF built-in httpd is interesting, but it seems like too much work to port to that now.

I'm currently trying to port my project to use cwhttpd and frogfs with IDF 5.0.3 and I'm running into quite a few compile issues. I'll take a look at your "next" branch.

BTW, I also want to factor out all the application specific parts like WiFi and OTA/flash b/c those are difficult to port between many version of ESP-IDF. Those can be in some other git modules or can just be in the example project(s). The Wifi and OTA are just a pain to maintain, and are too application specific to belong in the core httpd. Any objections to this?

[phatpaul]

Some more thoughts: We should document which platforms that the project should compile/work on, i.e.	Supported Targets	ESP32	ESP32-C2	ESP32-S2	ESP32-S3
ESP-IDF 3.x	❌ no support	-	-	-
ESP-IDF 4.4.x	❌ PR welcome!	?	?	?
ESP-IDF 5.0.x	✔	?	?	?
ESP-IDF 5.1.x	WIP (IDF51 branch)	?	?	?

This should help manage expectations and improve the new user experience.

On another project, I have played with some CI actions to build on various platforms and report build status. I could try that here. See: https://github.com/phatpaul/NMEA2000_switchbank_example_esp-idf

[jkent]

Glad to hear you are active again @jkent I like the progress that you have made on the project and I'm ok with the name change, as long as we agree to go in the same direction.

I'm not sure if I like the name or not. I'm open for better names. I just wanted to pick something that pulled it away from being espressif specific, since it can be embedded in Linux binaries as well. That's the direction I'm going with FrogFS.

I'm currently trying to port my project to use cwhttpd and frogfs with IDF 5.0.3 and I'm running into quite a few compile issues. I'll take a look at your "next" branch.

Please do, master is in an ugly state. Keep in mind though, I will be force pushing to next. Issues also all relate to next. Also, the httpd example in next uses the espressif httpd with the vfs interface. Standalone is a Linux binary.

BTW, I also want to factor out all the application specific parts like WiFi and OTA/flash b/c those are difficult to port between many version of ESP-IDF. Those can be in some other git modules or can just be in the example project(s). The Wifi and OTA are just a pain to maintain, and are too application specific to belong in the core httpd. Any objections to this?

Totally fine by me. I think making them add-ons is a good idea.

[chmorgan]

What was the proposed new name?

On Fri, Sep 1, 2023 at 3:40 PM Jeff Kent @.***> wrote:

Glad to hear you are active again @jkent https://github.com/jkent I like the progress that you have made on the project and I'm ok with the name change, as long as we agree to go in the same direction.

I'm not sure if I like the name or not. I'm open for better names. I just wanted to pick something that pulled it away from being espressif specific, since it can be embedded in Linux binaries as well. That's the direction I'm going with FrogFS.

I'm currently trying to port my project to use cwhttpd and frogfs with IDF 5.0.3 and I'm running into quite a few compile issues. I'll take a look at your "next" branch.

Please do, master is in an ugly state. Keep in mind though, I will be force pushing to next. Issues also all relate to next. Also, the httpd example in next uses the espressif httpd with the vfs interface. Standalone is a Linux binary.

BTW, I also want to factor out all the application specific parts like WiFi and OTA/flash b/c those are difficult to port between many version of ESP-IDF. Those can be in some other git modules or can just be in the example project(s). The Wifi and OTA are just a pain to maintain, and are too application specific to belong in the core httpd. Any objections to this?

Totally fine by me. I think making them add-ons is a good idea.

— Reply to this email directly, view it on GitHub https://github.com/chmorgan/libesphttpd/issues/104#issuecomment-1703242457, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJH4AANLXFM7EJHPY6AMRLXYI233ANCNFSM6AAAAAA4HUTOU4 . You are receiving this because you were mentioned.Message ID: @.***>

[jkent]

My fork of libesphttpd was rebranded as "Clockwise HTTPd" and had a symbol rename to cwhttpd_*

[phatpaul]

IMO this project is tied close to the espressif platform. I'm sure it could be refactored to be general-use, but there are plenty of other http server implementations out there which are probably faster, safer, cleaner, etc. This one just worked out of the box on ESP32 and had just the components I needed to start my project.

I think "esphttpd" would be a decent compromise, and has name recognition. It is at least nicer name than libesphttpd. But I'm ok with cwhttpd as well. I just don't want to have to change the name all over my code.

[jkent]

I did a lot of doxygen/breathe/sphinx/rtd documentation as well

[jkent]

IMO this project is tied close to the espressif platform.

Not true, it uses standard socket interfaces now. All the legacy esp8266 code has been pretty much eliminated. If esp8266 support is needed, I recommend using the ESP8266_RTOS_SDK which has the same socket interface.

Edit: I want to interject that I did most of the work on the fork as a Linux binary to speed up testing and development.

I'm sure it could be refactored to be general-use, but there are plenty of other http server implementations out there which are probably faster, safer, cleaner, etc. This one just worked out of the box on ESP32 and had just the components I needed to start my project.

I think "esphttpd" would be a decent compromise, and has name recognition. It is at least nicer name than libesphttpd. But I'm ok with cwhttpd as well. I just don't want to have to change the name all over my code.

I agree. This is partly why I consider my fork 'hostile'.

[chmorgan]

On the rename, can’t we have a legacy header file that maps one to the other via a whole mess of defines? Assuming the api is basically the same.

On the rename, is there a deeper meeting behind the name?

What makes the implementation unique? Small size? Embedded target?

On Fri, Sep 1, 2023 at 5:11 PM Jeff Kent @.***> wrote:

IMO this project is tied close to the espressif platform.

Not true, it uses standard socket interfaces now. All the legacy esp8266 code has been pretty much eliminated. If esp8266 support is needed, I recommend using the ESP8266_RTOS_SDK which has the same socket interface.

I'm sure it could be refactored to be general-use, but there are plenty of other http server implementations out there which are probably faster, safer, cleaner, etc. This one just worked out of the box on ESP32 and had just the components I needed to start my project.

I think "esphttpd" would be a decent compromise, and has name recognition. It is at least nicer name than libesphttpd. But I'm ok with cwhttpd as well. I just don't want to have to change the name all over my code.

I agree. This is partly why I consider my fork 'hostile'.

— Reply to this email directly, view it on GitHub https://github.com/chmorgan/libesphttpd/issues/104#issuecomment-1703323544, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJH4AB7QANDPNGE4MER5HTXYJFPVANCNFSM6AAAAAA4HUTOU4 . You are receiving this because you were mentioned.Message ID: @.***>

[jkent]

On the rename, can’t we have a legacy header file that maps one to the other via a whole mess of defines? Assuming the api is basically the same.

I'll have to look, thats an idea. I've been removed from the project for some time so I don't recall what all I've done.

On the rename, is there a deeper meeting behind the name?

I don't really know how I came up with it. I guess because its progressive development.

What makes the implementation unique? Small size? Embedded target?

Easy to define and implement routes with an interface that doesn't get in your way. That's the huge thing for me.

[phatpaul]

I'm interested in a stable library and reference project that I can use as a basis for projects with a low barrier to entry.
I think we should strive for a relatively stable interface and support for the latest 2 or 3 "stable" releases of the espressif environment (at this time it's 5.1, 5.0, and 4.4). The example projects should compile without errors in those environments unless stated in the readme. (this includes python versions too)

Right now, chmorgans fork compiles on 4.4. And I'm still working on getting jkent's "next" branch to compile on 5.1 or 5.0. But I'm out of time for this week. I'll revisit on Tues. Have a good long weekend guys.

[phatpaul]

@jkent FYI I was using this example and managed to fix some things, but still stuck with a few build errors. https://github.com/phatpaul/cwhttpd-example If you have a chance would you please see if you can get that project to build with IDF 5.1. Or should I be using your other example in the frogfs/examples/httpd/?

[jkent]

cwhttpd-example is at least a year out of date. frogfs/examples/httpd is what you should be using as it reflects current state of things. It uses the espressif http server though. Not cwhttpd/esphttpd

I've been developing on Linux (debian bookworm) I have yet to test my work under Windows. It's on my todo list.

[jkent]

Hey @phatpaul

I've worked out all the issues so FrogFS should work fine on a Debian buster machine with IDF 5.0 and 5.1. I'm not sure if it is possible to support 4.x because you can't have conditional component dependencies, and espressif split spi_flash -> esp_partition

--- much later --- Just finished tweaking and testing Windows support. The standalone binary needs work (I don't know my way around cl.exe) but the httpd demo builds now.

[phatpaul]

Thanks @jkent that's great progress. I'm still trying to get cwhttpd + frogfs working by updating this old example: https://github.com/jkent/cwhttpd-example/pull/2 Now, I'm hitting a link error and it looks like components/frogfs/src/route.c is not being compiled at all.

undefined reference to `frogfs_route_get'
undefined reference to `frogfs_route_index'

I don't think it makes sense for route.c to be part of the frogfs component, since it is just a route handler for cwhttpd (which depends on frogfs). That would require frogfs --depends on--> cwhttpd.
Wouldn't it make more sense to move that file to the cwhttpd component and add an optional dependency (cwhttpd-- depends on--> frogfs)?

[jkent]

Yeah, I agree with moving it to cwhttpd. I probably didn't do that because I didn't want the hard dependency on frogfs. But link time optimization is our friend here.

[jkent]

I'll work on finishing up the httpd demo (using the espressif component) tonight, and then I'll work on cwhttpd. What I did today to FrogFS:

• Removed the main sort table
• Made directory objects have a sort table of child objects
• Rewrote directory handling code
• Rewrote mkfrogfs.py
• Updated cmake code to reflect this
• Confirmed building with and without directory support works

I think I'm done breaking binary compatibility with FrogFS in the next branch for a while. Feedback is welcome.

[jkent]

Oh another huge thing I snuck in there too, which i forgot to mention in the commit, because I forgot, file data is now aligned instead of packed, so files can be mapped aligned to a struct when using frogfs_access()