Replacement for Civetweb

[cfriedt]

Introduction

This (previously an RFC) is intended to be discussed by network subsystem maintainers, the TSC, and existing users of the Civetweb module.

The replacement for Civetweb must be ready for LTSv3 (Civetweb was available in LTSv2, and we should have a suitable replacement for LTSv3). However, ideally, this would be completed as soon as v3.3.

Comments & corrections are welcome.

Problem description

Civetweb has been a bit of a pain point for a while due to lack of regular module maintainership. There is also a minor technical shortcoming in that it does not directly support TLS.

The module has been deprecated (https://github.com/zephyrproject-rtos/zephyr/pull/46061) and likely will be removed shortly (e.g. https://github.com/zephyrproject-rtos/zephyr/pull/46746).

However, HTTP is fairly important - some might even say it should be a first-class citizen - not simply for demo purposes, but also for production systems.

With that in mind, we may want to consider providing a suitable alternative to support the following sample applications:

• http_server
• websocket_server

In order to maintain feature parity with Civetweb (as supported within Zephyr) the requirements for a replacement HTTP server are:

• Permissively licensed (Apache-2.0, BSD*, MIT, ..)
• Optimized for size (both code space and memory usage)
• Support for HTTP/1.1
• Support for operating without a filesystem
• Easy extensible to add custom URIs and handlers (e.g. REST)
• WebSocket server support
• Ability to use e.g. Zephyr's JSON library
• Protection against any exploits / CVEs previously met with Civetweb
• IPv4 / IPv6 Support
• Reporting statistics e.g. vis JSON

CVE details and Additional "nice to have" features are discussed in Detailed RFC.

Proposed change

This RFC proposes to adopt one of the two following options:

1. External Zephyr module: Adopt Merecat (fork of thttpd)
2. Internal Zephyr library: Design an HTTP server

Detailed RFC

Additional Details

Aside from the features required for parity, there are a number of "nice to have" features for a replacement HTTP server.

• Support for TLS
  • Note: Zephyr implementation may be in flux (https://github.com/zephyrproject-rtos/zephyr/issues/43712)
• Configurable via Kconfig (filesystem, maximum number of clients, timeouts, buffer sizes, features, ..)
• Built with CMake (enable / disable compiling sources based on feature set)
• Decent performance (scales well with number of clients, requests)
• Secure (should not be a regular source of CVEs)
• Easily maintainable (SW architected well)
• Easily testable (via ztest, if possible)
• Support Compressed Resources in Memory (CRiMe)
• Support filesystem resources
• A web-based dashboard (e.g. via REST / JSON)
• HTTP/3 (eventually..)
• HTTP Client / WebSockets Client - quite useful for testing, but could be made available to the application

Previously discovered exploits in Civetweb that have affected Zephyr are listed below:

• TODO

Comparisons with other Web Servers

There are some readily available comparisons of various HTTP servers. E.g. Wikipedia, thttpd site. Mongoose can probably be swapped with Civetweb for comparison purposes, and the same for thttpd and Merecat.

Tabular comparison

Note: - should be interpreted as "not yet but eventually yes".

Feature	Civetweb	Merecat	Custom
License	MIT	BSD-2-clause	Apache
Size?	Optimized	Optimized	-
HTTP/1.1	Yes	Yes	-
No FS	Yes	Yes	-
REST	Yes	Yes	-
WebSockets	Yes	No	-
JSON	Yes	No	Yes
CVEs	Yes	No	Yes
v4/v6	Yes	Yes	Yes
Stats	Yes	?	Yes
TLS	No?	(OpenSSL)	-
X.509	No?	?	-
Kconfig	No	No	Yes
CMake	No	No	Yes
Perf	No?	Yes	-
Secure	Yes?	Yes	-
Authentication	?	Yes	-
Maintainable	No	No	Yes
Testable	No	No	Yes
gzip / deflate	No	(Zlib)	-
FS	No	Yes	Yes
Dash	Yes	No	Yes (JSON)
OAuth	?	?	-
SASL	?	?	-
HTTP/3	No	No	(Eventually..)
-------	----	----	-------
SCORE	11	11	12-20

TLDR / Summary

• Obviously, no need for CGI at this time
• HTTP Authentication is ~OK, but a TLS / mutual auth / X.509 would be much better
• Making an HTTP server requires some dedication
  • as pointed out, writing a simple HTTP server is easy, but writing a robust one, not quite as easy
  • would need to be put on the roadmap
  • not entirely within scope of responsibilities of Network Maintainer (overloaded already)
  • would need to volunteers for maintainers / collaborators
  • would need to have someone from the Security WG involved regularly
  • can use TDD to get up to speed
  • tests to protect against previously encountered CVEs
• Zlib is probably not suitable to include in Zephyr
  • deflate / gzip is luckily not very difficult to implement (https://github.com/zephyrproject-rtos/gsoc-2022-thrift/issues/91)
  • some resources can be compressed offline, at build time. stored in ROM, no runtime requirements
  • if supporting FS resources and compression, then possibly deflate / gzip as a runtime requirement
• A web-based dashboard would be nice, although both could be configured
• Part of the issue that made Civetweb difficult to maintain was that it was a module
  • Doing the west.yml / revision pinning dance can slow things down a bit
  • Keeping something in-tree could have faster turnaround for bugfixes & testing
• We likely want to use a typically single-threaded ("select") model to reduce resource utilization
  • It may be possible to have some configurable number of worker threads
• Merecat is attractive, but would still require adding support for a number of features:
  • WebSockets
  • mbedTLS
  • compressed resources in ROM
  • Kconfig
  • Tests
  • ...

Proposed change (Detailed)

It does not make a lot of sense to go into detail about adding another HTTP module to Zephyr that simply tracks a third-party repository. We have done that before. It is an option.

Details of focus on proposal of purpose-built HTTP solution for Zephyr

General Design

• Top-down design with Kconfig
  • maximum timeouts / keep-alive
  • maximum buffer sizes
• Test-driven development with C++, POSIX API
  • Functional first, optimize later
  • POSIX API allows for native development
  • Use of STL for ease of bringup (can be swapped later if necessary)
• HTTP is a string-heavy protocol => make heavy use of std::string_view
  • string_view: (pointer, size) for constant strings
  • C++17
• Make use of literals / constexpr as much as possible
  • HTTP Headers: key-value pairs => unordered_map<string_view, string_view>
  • Request Headers held constant while servicing request
  • Response Headers held constant while servicing response
• Resources via unordered_map<string_view, resource_obj>
• Support FS via CONFIG_POSIX_FS
• Initially, single-threaded design ("select" approach)
  • Consider multi-threaded later, if necessary

Compressed Resources in Memory (CRiMe)

• Memory is ROM in this case
• Quite straightforward at build time with CMake and a python script
• For each resource in CONFIG_HTTP_WEB_ROOT="/path/to/dir"
  • GZip / Zip resource
  • Convert to C array with xxd / python
  • Generate metadata, C/C++ source, including array (python script)
  • Generate CMake target

REST Support

• Endpoints declared in application C/C++ (GET, SET, POST, etc)
• Statically registered via iterable section?
• Endpoints really just filling in callbacks for supported methods

Filesystem Support

• depend on CONFIG_POSIX_FS
• should allow users to drop files into e.g. /www on an SD card and immediately serve them as resources
  • need FS "event listener" API (does that already exist?)
• a concrete use-case for optionally supporting runtime compression of served resources
• If a URI contained in the FS (/www) conflicts with a CRiMe object, which has priority?

URI Priority

1. REST endpoint
2. CRiMe
3. FS (if so configured)

Ordering configurable? It may be desireable to have FS resources override CRiMe 🤷

Dependencies

Both options can be implemented in such a way that they do not have external dependencies.

Concerns and Unresolved Questions

• Complexity w.r.t. building and maintaining a good embedded web server
• Complexity about maintaining essentially a fork of Merecat (if that is something we consider)
• Anxiety about another external HTTP module
• How best to integrate TLS?

Alternatives

See previous comparisons

Action Items

• [x] https://github.com/zephyrproject-rtos/zephyr/pull/51141
• [x] https://github.com/zephyrproject-rtos/zephyr/pull/55430

[cfriedt]

Just putting this out there for some early feedback, since we have a PR to remove civetweb already.

@mbolivar, @beriberikix, @rlubos

I'm not really sure which option is better, TBH. An HTTP server as a module has been problematic in the past, but a custom one could be a fair amount of work with all of the "nice to have" features. OTOH, an HTTP server that has been around for a long time has been vetted, but if a custom one were created it could be tuned to work with Zephyr quite well, tests could be added for everything from functionality to CVEs to performance.

If there are alternatives that are better than Merecat, it would be great to have them added to the table.

[cfriedt]

Approximate state machine for single-threaded "select" model (really uses poll)

[beriberikix]

Here's a use case that might help in scoping - browser-based device setup. Common for Wi-Fi consumer IoT product, the typical flow is the product tells the consumer to look for the device's local network, connect to an IP or DNS-SD address and enter their network credentials (loosely described here.)

The requirements are fairly modest as the server tends to be simple, typically with minimal static HTML and CSS embedded into the webpage and some way to interact with the OS (ex. store credentials via the Settings subsystem.) Often times these pages are not encrypted but being able to serve them via HTTPS is recommended.

[beriberikix]

Additional ideas:

Would it make sense to support an optional FS or other way to serve binary assets? For developer convenience I can imagine someone wanting a way to upload assets like HTML, CSS, JS & images into a "directory" and simply point the server to it ala /www.

Adding a shell would be a nice-to-have, like the ability to start/stop/clear cache, etc.

CBOR for the RESTful aspects would be a nice-to-have.

Other typical RESTful API features, like authentication (OAuth, SASL) should be explicitly out of scope but noted somewhere in the RFC.

CivetWeb also includes an HTTP client - is that in scope or needed?

X.509 in scope?

[cfriedt]

@beriberikix - generally agree with every point you have made, and ❤️ the idea of browser-based device setup / provisioning.

Would it make sense to support an optional FS or other way to serve binary assets? For developer convenience I can imagine someone wanting a way to upload assets like HTML, CSS, JS & images into a "directory" and simply point the server to it ala /www.

Absolutely - I'll elaborate on the FS option in the RFC.

Adding a shell would be a nice-to-have, like the ability to start/stop/clear cache, etc.

CBOR for the RESTful aspects would be a nice-to-have.

Shell would be nice, but we could also use the browser for that. Mentioned already, but both of those options could be made available via Kconfig - i.e. either web, shell, or both.

Other typical RESTful API features, like authentication (OAuth, SASL) should be explicitly out of scope but noted somewhere in the RFC.

OAuth for sure - thanks for mentioning that. I think there must be a library available that we could pull in.

CivetWeb also includes an HTTP client - is that in scope or needed?

Mentioned as well - I originally thought that was out of scope, because it isn't part of a sample application. However, an HTTP client would probably be smart even to just make the testsuite efficient, so I think there is a definite argument for it.

X.509 in scope?

For sure - I used the term mutual authentication, but X.509 is better.

Will adjust the RFC - thank you for the input!

[carlescufi]

Test-driven development with C++, POSIX API C++17

So far we have not included any Zephyr-specific code that is written in C++. I am not convinced this is a good idea and, in any case, it would require acceptance at the TSC level. While I completely understand how C++17 can help develop something like a web server, I'd rather see C++ bindings to a C-written server to avoid imposing C++ on anyone.

[rlubos]

CivetWeb also includes an HTTP client - is that in scope or needed?

Mentioned as well - I originally thought that was out of scope, because it isn't part of a sample application. However, an HTTP client would probably be smart even to just make the testsuite efficient, so I think there is a definite argument for it.

Note that Zephyr already has a simple HTTP client library: https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/net/lib/http/http_client.c

How best to integrate TLS?

Do you plan to use TLS sockets? Any concerns at this point?

[cfriedt]

Test-driven development with C++, POSIX API C++17

So far we have not included any Zephyr-specific code that is written in C++. I am not convinced this is a good idea and, in any case, it would require acceptance at the TSC level. While I completely understand how C++17 can help develop something like a web server, I'd rather see C++ bindings to a C-written server to avoid imposing C++ on anyone.

@carlescufi My main motivation for using C++ would just be for quick bring up (natively). Given that the lingua franca of Zephyr is C, the API bindings for a custom HTTP impl would also most certainly have to be in C.

[undisputed-seraphim]

Why not just complete the POSIX networking API and let existing network libraries do the rest of the work? I've been trying to port a Boost.ASIO-based application to Zephyr and it's mostly just missing some POSIX APIs and defines. I wrote a hacky, not-for-production-use shim to paper over the missing APIs and it mostly works. Not saying everyone should do this, I was just trying to demo stuff.

[cfriedt]

Note that Zephyr already has a simple HTTP client library: https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/net/lib/http/http_client.c

Perfect - would be great to leverage that :-)

How best to integrate TLS?

Do you plan to use TLS sockets? Any concerns at this point?

I think it makes sense to use whatever TLS implementation the project intends to support, and there may have been talk of changing that IIRC.

Edit: I checked and it was specifically talk about adopting the PSA Crypto API

[cfriedt]

Why not just complete the POSIX networking API and let existing network libraries do the rest of the work? I've been trying to port a Boost.ASIO-based application to Zephyr and it's mostly just missing some POSIX APIs and defines. I wrote a hacky, not-for-production-use shim to paper over the missing APIs and it mostly works. Not saying everyone should do this, I was just trying to demo stuff.

At the dev review (or what is the TSC meeting?) that was more or less the agreed direction. I.e. rather than go through the exercise of trying to integrate a 3rd-party server, simply tailor one precisely to Zephyr's needs and leverage existing code.

[nvincent-vossloh]

Hi, I recently upgraded to zephyr 3.1 and noticed the deprecation warning on civetweb which we use in our product in a similar fashion as described here (first time setup, configuration, ...).

I wanted to let you know that we have modified civetweb to support https. If it can help anyone... I'm really sorry I did not took the time to make a proper PR, let me know if it is worth it.

[bonetor]

Are there any updates for the replacement of CivetWeb?

@nvincent-vossloh I took a look onto your modified CivetWeb. Where / how do you handle TLS?

[cfriedt]

Notes from Network Forum 2022-07-26:

• collect stakeholders (ideally, companies who would benefit significantly from an HTTP server)
• collect collaborators - @cfriedt, @beriberikix?
• maintainer?? @cfriedt? co-maintainers? Interested parties apply

This cannot be a free-time thing (HTTP is fairly critical for an IoT RTOS), and we would prefer to have some company support.

[cfriedt]

Additional Notes from Network Forum 2022-07-26:

• PSA Crypto API - standard crypto API for Zephyr
  • unified api for all crypto apps
• @cfriedt could come up with initial implementation / PoC, with testsuite
  • Would be greatly appreciated for interested parties to contribute test-points / cases to testsuite
  • Additional HTTP features should mainly be additive changes (i.e. SW architected to not require major redesign for new features)
  • This includes HTTPS (although HTTPS is 1st-class feature that will definitely shape architecture)

[cfriedt]

Are there any updates for the replacement of CivetWeb?

There are no updates yet.

[nvincent-vossloh]

@nvincent-vossloh I took a look onto your modified CivetWeb. Where / how do you handle TLS?

The civetweb modifications for TLS on zephyr are on our civetweb fork https://github.com/SiemaApplications/civetweb/commits/zephyr

The instantiation of the server using TLS is in a private repository, so obviously I cannot give you the exact code. However the commit which introduced https instead of http is atomic and should be easily understandable : civetweb_tls_instance.diff.zip, I have removed the private key from the diff, even though they are for dev purposes only. With the civetweb samples available in samples/net/civetweb you should be able to figure out everything to instantiate the server with a TLS connection.

[KozhinovAlexander]

Hello everyone,

Thank you all for pointing on this topic. I was previously a maintainer for Civetweb and there was two major problems with it:

1. Huge RAM consumption compared with embedded devices.
2. Long long long Integration processes in Zephyr itself, since no one had time to take a look over it.

But! I've begun working on my own WebSocket Server Implementation for Zephyr. You can take a look at websocket_server branch in my Zephyr fork. Here what already works:

1. Opening handshake over http. Https can be easily added, thanks to great samples from zephyr.
2. Receiving non-framed packets over Websocket protocol.
3. Answering to the Websocket packets is mostly done, but still in progress.

There are still many todo's also marked in code by me. There are s some great and some suboptimal SW architecture decisions. But! Everybody wanting to participate is welcome to pull request in my fork. At the end it will flow to zephyr under apache license anyway.

Any feedback is welcome.

[cfriedt]

Thank you all for pointing on this topic. I was previously a maintainer for Civetweb

@Nukersson - would you be interested in being a collaborator or perhaps maintainer for a purpose-built HTTP server? I cannot make this a priority with work atm, although I think it's quite important and have some interest in seeing it happen.

I'll check out your websocket_server_api, branch as well. There is a link below that can be of use for anyone interested. https://github.com/zephyrproject-rtos/zephyr/compare/main...Nukersson:zephyr:websocket_server_api

[KozhinovAlexander]

Thank you all for pointing on this topic. I was previously a maintainer for Civetweb

@Nukersson - would you be interested in being a collaborator or perhaps maintainer for a purpose-built HTTP server? I cannot make this a priority with work atm, although I think it's quite important and have some interest in seeing it happen.

I'll check out your websocket_server_api, branch as well. There is a link below that can be of use for anyone interested. https://github.com/zephyrproject-rtos/zephyr/compare/main...Nukersson:zephyr:websocket_server_api

Hello nad thank you.

I am interested in collaboration to http server dev. But maintaining it will be for me currently too much due to private reasons.

Thank you l.

[bonetor]

Hello. We are using CivetWeb since Zephyr v2.7 and

• would be highly interested in a replacement of CivetWeb
• glad if there will be a successor for CivetWeb

With Zephyr v3.2 the support and maintenance of CivetWeb is stopped and there is currently no support for an alternative HTTP web server with similar functionality as CivetWeb: REST API, WebSockets, HTTPS / TLS / Secure WebSockets, ...

[cfriedt]

Some additional notes from the Networking Forum 2022-10-04 https://bit.ly/3ruOUWX

[madscientist159]

Was fairly disappointed to see Civetweb disappear in 3.2, as it is a requirement for the Kestrel BMC. Currently stuck on 2.x / 3.1.x until a Web stack is available again.

We need HTTP and websockets for the BMC functionality.

[madscientist159]

For anyone needing a temporary fix to get their Web applications running on Zephyr 3.2.x in the interim, this patch reenables the integration. Tested to work on ppc64el. :christmas_tree: :gift:

https://gitlab.raptorengineering.com/kestrel-collaboration/kestrel-firmware/zephyr-rtos/-/commit/008cbf2a20a82e5187a892885f1bff72695345c4

[cfriedt]

This seems like a decent starting point for HTTP/2 conformance.

https://github.com/summerwind/h2spec

There really are not a lot of freely available http conformance Test suites.

[cfriedt]

Heh... just thought of something. This would make a great GSoC project 😁

[cfriedt]

Copying / pasting from Discord:

GSoC Project Idea: Replacement for CivetWeb + Conformance / Test Suites

The main benefit to the Zephyr community is that we would have a Designed-For-Zephyr (TM) scalable HTTP server that is maintained in-tree. Part of the challenge with CivetWeb is that it was maintained separately as a module that needed a lot of attention and integration work with the upstream CivetWeb.

Designed-for-Zephyr has a lot of interesting implications:

• compile-time const data could help reduce memory consumption (e.g. iterable sections, const configuration, ..)
• zephyr-specific processes could be used for running sanitizers, linters, etc
• consistent coding style, built-in documentation
• synchronization and maintenance becomes much easier than with an external module / project

Aside from transposing and subsequently passing conformance tests for HTTP, extra challenges would be

• Adding protection against previous CVEs that affected CivetWeb on Zephyr
• Being cognizant to write secure code and tests so as to avoid introducing security issues
• Getting to feature-parity with the previous CivetWeb module
• Adding Instrumentation features, so that it's easy to extract required metrics
• Optimizing for both space and speed, ideally
• Keeping things flexible / future proof (e.g. HTTP 1.0, 1.1, 2.0, 3.0, ..)

There are several resources that can be used - e.g. Zephyr's JSON, http client, and websocket client libraries, mvedTLS, ZIP implementation, etc. I would most likely provide the skeleton like what was done with Thrift.

And it should go without saying, bragging rights would be pretty substantial, writing the HTTP server used on a 1st-class IoT RTOS like Zephyr.

Mentor(s):

• @cfriedt / Meta
• Would be great to have a co-mentor from another company who might have an interest in such a thing.

[cfriedt]

Added to the project wiki https://wiki.linuxfoundation.org/gsoc/2023-gsoc-zephyr

[jgl-meta]

@cfriedt Can you provide a brief status update for this? Will it be ready for the 3.4 release?

[cfriedt]

@cfriedt Can you provide a brief status update for this? Will it be ready for the 3.4 release?

@jgl-meta - Judging by the wiki and by the GSoC timeline, it will not be ready. Please push it to 3.5

https://github.com/zephyrproject-rtos/zephyr/wiki/Release-Management https://developers.google.com/open-source/gsoc/timeline

[jaskij]

Just to add to the list of considerations: gRPC. While not entirely on-topic, it does use HTTP/2 as a transport, so would presumably need at least some considerations from the HTTP server implementation.

Link to specs: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md

[henrikbrixandersen]

@cfriedt Can you provide a brief status update for this? Will it be ready for the 3.4 release?

@jgl-meta - Judging by the wiki and by the GSoC timeline, it will not be ready. Please push it to 3.5

@cfriedt Will this be ready for v3.6.0?

[cfriedt]

@cfriedt Will this be ready for v3.6.0?

Maybe?

I would like to see if @Emna-Rekik would like to fix some of the integration issues.

I have little to no bandwidth to do this in the immediate future.

Given that 3.6.0 should be released in February, I could maybe try to find some time for it during the holidays, while I'm supposed to not work and spend more time with family.

[henrikbrixandersen]

Maybe?

I would like to see if @Emna-Rekik would like to fix some of the integration issues.

I have little to no bandwidth to do this in the immediate future.

Given that 3.6.0 should be released in February, I could maybe try to find some time for it during the holidays, while I'm supposed to not work and spend more time with family.

I am only asking since we keep listing this as one of the goals for a given release, but we keep missing it and pushing it to the next release. Another option could be to just remove the milestone altogether and just get it in once it is ready.

[jukkar]

Given that 3.6.0 should be released in February, I could maybe try to find some time for it during the holidays, while I'm supposed to not work and spend more time with family.

I am only asking since we keep listing this as one of the goals for a given release, but we keep missing it and pushing it to the next release. Another option could be to just remove the milestone altogether and just get it in once it is ready.

I suggest that holidays are not used for this and we remove the milestone. It is ready when it is ready.

[cpq]

Hi everyone, I represent cesanta/mongoose. For those who are not aware, this is the original project civetweb was forked off. Also, I am an original author of Mongoose - thus, of civetweb as well.

My team develops Mongoose for many years. As you could see from the Github repo, it is an active project, and we make efforts to keep it issue-free. We would be happy to maintain Mongoose for Zephyr, and guarantee almost-zero ongoing issues with it, like we do for the Mongoose project itself. We'd be happy to provide support for users, too. Mongoose has everything an IoT developer might need - including professional device dashboard with OTA support, etc. It is our bread and butter to provide a state of the art embedded web server starting from 2004. One of the upcoming features will be native built-in TLS 1.3 support, optimised for low RAM usage and quick handshake.

However, Mongoose is dual licensed: GPLv2/commercial. It is not BSD/MIT/Apache.

Is that a show stopper? Would it be possible to have Mongoose as an alternative for those who choose to use it and OK with the license?

[rlubos]

Hello @cpq,

Unfortunately, I'm afraid GPLv2 license is a no-go for integrating a project with Zephyr, even as a module due to license incompatibility (you can check discussion on the topic in https://github.com/zephyrproject-rtos/zephyr/issues/55861 for more details).

[cfriedt]

I think I'm going to close this, since the GSoC and additional support seems to have pushed this past the finish line.

Thanks for the additional effort by @rlubos, @jukkar @pdgendt and others 🙏