[Feature Request]: nRF52 ethernet http(s) server integration

[qjayjayp]

Platform

NRF52

Description

For my home station I would like to not need to rely on wifi (which for me drops off some times, and I have other constraints with it (namely a "not so good" router that is the nearest and energy saving needs), but instead use the wired ethernet connection via the RAK shield (with its WIZnet W5100S).

I had a bit of a look on why it's not already supported given that some connectivity (e.g. mqtt and some more) do already work. Again this (platformio, nRF52* stuff) is not my special topic, but I hope I can trigger some initial thoughts and ask some questions, please correct me in case I took the wrong path somewhere, found the wrong library dependency or something:

• ESP32 is using the https://github.com/fhessel/esp32_https_server which is already copied into the meshtastic project (so maybe it got some meshtastic specific adjustments?
• Now the mesh/http code (which uses the above https_server) seems to use EPS32 WiFi classes (https://github.com/arduino-libraries/WiFi/), which in the end use some SPI driver (so is it that ESP32 has it's wifi module connected that way?)
• What I don't understand here is that esp32_https_server though uses the classic posix/linux networking APIs (socket, bind, listen, accept, recv, send), so am I wrong here or does the ESP32 framework has those supported accordingly?
• If I look at the nRF52 (e.g. MQTT) implementation it uses the according RAK library which also goes down to SPI to connect to the WIZnet shield (https://github.com/RAKWireless/RAK13800-W5100S), fine so far.
• The RAK Ethernet library uses some proprietary, but similar (as it's still networking) functions that look like they could be mapped to the posix/linux ones (in case that's relevant):
  • socket (...socketBegin())
  • bind, listen, accept (...socketConnect())
  • recv (...socketRecv())
  • send(...socketSend())
• Now what looks more difficult to me is that the esp32_https_server has two dependencies on mbedtls and openssl which seem to be provided by the ESP32 framework but I could not see a lot of people sing them on nRF52 (although there where). Maybe the already crypto already used in nRF52 can/needs to be adopted for mbedtls, but I don't know yet what is really required.
• So in case I'm not on a completely wrong road already I would like to ask whether it makes sense to look for integration of mbedtls and openssl to build the esp32_https_server for the nRF52 platform or that's nonsense (for dependencies, size or other reasons)?

[garthvh]

There is no web server available for nrf currently.

[RicInNewMexico]

The issue with the W5100S ethernet interface is that it is what's opening the socket, listening, accepting, etc.. not the firmware running on the node. The W5100S adds a layer of abstraction.

When you see the socketConnect() call being made that is an instruction to the W5100S's onboard controller to open a socket. The entire actual TCP/IP "stack" is running on the W5100S and not implemented in the firmware.

The W5100S is capable of acting as a very rudimentary http server or http client, can function as a raw TCP server or client or do basic UDP unicast / multicast / broadcast, etc. The HTTP functionality doesn't have any direct access to firmware / filesystem from which to serve html / css / etc. The functionality that it does have is more or less serving up, for example, pin status in unformatted plaintext. It lacks the necessary functionality to serve anything that's stored external to the ethernet controller.

[qjayjayp]

There is also a ready made but http only library (https://github.com/khoih-prog/EthernetWebServer) that supports the WIZnet shield and also some non-RAK nRF52 board. While for my local use case this would be sufficient, it feels wrong to block out https, as there are surely other uses cases which need it.

[qjayjayp]

@ both of you, so you are saying that the W5100S only has client functionality, but no API for a server (I actually forgot to mention above that meant socketListen() for the posix listen() equivalent, although I don't know whether this is an actual equivalent or not, according to what you say it isn't)

[garthvh]

There is no web server at all, it allows for a TCP connection to the device and for MQTT to work. There is no on device webui available as it has no web server.

[qjayjayp]

Hi Garth,

my initial request was maybe not so clear (I think I jumped a bit too far for the beginning), I meant that the request should be about adding an webserver that is currently not available.

On the other hand I tried the quickest and ugliest solution to my actual problem (using the self hosted website via a meshtastic device, which in this case shall be a RAK4631 with the ethernet shield) and I have now created some simple code that can answer port 80 queries to /api/v1/fromradio, so extending on that I should be able to get what I need, but the broader question here was whether this should be taken care of by the experts and made available to all people.

Regards,

Jörg

[garthvh]

You can already use the hosted version with the Ethernet module, still not sure why there would be a need to change any firmware code.

[qjayjayp]

If I flash stock firmware and debug the website it says connection refused for all requests that go to the device' IP (it's pingable at the same time, I'm currently using DHCP). Btw, I found interesting that the wizNet in listening mode is not detectable by nmap (port shows as closed), but a nc (netcat) on the port can still open it (and if a response only is programmed that can be read by netcat).

The website also has the need (dont' know where this comes from) that it needs to retrieve the page /hotspot-detect.html only after I replyed that with 404 it continued on with radio requests back and forth.

[qjayjayp]

Also for debugging I used some code from the RAK library that shows the state of all sockets (you need to move some methods from private to public to run that code from somewhere else than the Ethernet... classes) and it only showed one port 80 listen (that I added), so I would be surprised if it works otherwise (also no port 443 is visible in listen state).

My code though works now to a degree to show the device state and config on the webpage initially, but it's somehow not getting updates later on (although it looks fine on the network and the device, as far as I can see it (but I need to look deeper here)), but I'm working on getting request reading more stable (currently it's only based client.available() in a loop and if my PC starts e.g. swapping so it's getting slower, this loop on the device is faster than my laptop can deliver the request, so only part of the request is received), also I will have some more questions (if anybody is willing to support), on the multithreading work mode, e.g. I see the Wifi WebServer using runonce() with a return value of 5ms, but also in my tries I locked up the whole device because of endless loops in runonce() so it seems to be sensitive and there are maybe some rules or best practices on how long you should take and then return to give others a chance (I'm assuming there is no real multi threading but calling all those runonce() methods according to a plan derived from their returned 'run next' times?) But I could be wrong, corrections welcome.

[garthvh]

Have you tried with the android app? TCP connection is already supported. I still don’t really get what problem you are trying to solve, and adding a webserver on nrf if going to be a fragile hack that uses a lot of resources, esp32 devices run twice as fast with WiFi on.

[qjayjayp]

No, but just tried it, connecting to an ESP device works, to the nRF doesn't (and it needs a bit of finding out the right combination, first enter IP address into IP field, then activate radio button called 'IP address' (IP gets added to the list of devices), if connection works IP is blank and info is loaded and viewed, if connection does not work IP stays greyed out).

[qjayjayp]

Well if you say resources (CPU power and/or RAM) are limited, that would be a point, but until now it's working surprisingly well already (well now I'm in a redesign phase where nothings works, but will get over it). If it's solvable this way, I wanted to change my ESP32 device that is used in a fixed installation at home, which shows the webpage for residents to interact with (the usual stuff, chat with people, watch the map), and is rebooted every day (shutdown at night), so I added an auto connect to the webpage, that means it shall be mostly unattended(!) and if somebody writes a message during the day, it shall be possible to view that message later on the webpage, which is not possible (message is missed by the webpage) if wifi is dropping off, so I was hoping for more TCP stability via cable (which is the case in general for all devices I have worked with so far, but I see there are some challenges here). I have less experience with embedded devices, but if resources really get scarce I can account for that no problem, I can imagine though that a full blown HTTP and HTTPs webserver could be difficult if it's really tight on resources. What I have now is not a webserver at all, it's the simplest code that can handle the needed HTTP requests, so a bit of parsing and passing data forward and backward.

[qjayjayp]

Btw, is the android app in IP mode also using http or a different protocol? Unfortunately my station is a linux PC that already shows a website where the meshtastic website is already integrated, so I'm currently not planing on changing devices there to android or so, in case a non-http protocol is there already. Is there a raw proto-buf protocol? (that would skip the http part)

[garthvh]

Http. Every connection type sends protobufs, you can use BLE, serial or http

[garthvh]

If you are wanting Ethernet with a webserver the S3 boards are probably the best choice, older esp32 boards do not have enough pins for both Lora and an Ethernet shield.

[qjayjayp]

... S3 ... ahhh, when initially thinking about the RAK and buying it, I was not aware of WizNet or it's interfacing at all, I thought of the RAK as a closed solution, but now with looking at the code and recognizing that the WizNet is just added to (virtually any) SoC via SPI and it doesn't even really look like the RAK WizNet library has any RAK specialities but is rather a WizNet to SPI lib (could be wrong here though, missing some details), it could be added to any SPI on other systems. I've seen the S3 has 4 SPI buses, where I work these are really used as a bus, so shared between multiple devices, but how is it with the meshtastic firmware, somewhere between the lines I read that dedicated SPI busses are used at some places (if I m not mistaken with some other projects now). I actuall have a Heltec Wireless Stick V3 (the one with the too small display) right now for the station (that should have 4 SPI busses even), so I guess it's possible to stick a WizNet shield to one of those. I will have a look, but maybe I need to configure the PINs accordingly so that everything is found. Thanks for giving that idea (about one week ago I would probably not have understood it (missing SPI/WizNet info), but now makes perfect sense)!

[David-Woodward]

Hi Garth,

my initial request was maybe not so clear (I think I jumped a bit too far for the beginning), I meant that the request should be about adding an webserver that is currently not available.

On the other hand I tried the quickest and ugliest solution to my actual problem (using the self hosted website via a meshtastic device, which in this case shall be a RAK4631 with the ethernet shield) and I have now created some simple code that can answer port 80 queries to /api/v1/fromradio, so extending on that I should be able to get what I need, but the broader question here was whether this should be taken care of by the experts and made available to all people.

Regards,

Jörg

Jörg, if you have time to share it, I'd be interested in taking a look at the code you used to connect the self hosted web-client to the RAK4631 over ethernet.