add HTTPS (SSL) to Web and TLS to MQTT

[proddy]

Secure connection

[proddy]

Planning to port AsyncWebServer to https://github.com/fhessel/esp32_https_server, which does self-signed certs and TLS

[proddy]

created new branch for this https://github.com/emsesp/EMS-ESP32/tree/ft_https

[proddy]

Security wasn't that important for most users so I'll stop working on the port.

[claymore666]

I would like that. I just discovered that the async client from Marvin already comes with SSL/TLS support. I am using EMS-ESP to populate my PostgreSQL Database via HiveMQ, and its the only device currently which cannot use SSL for transfer of the data. Thanks for a great product and your efforts.

[proddy]

cool, do you want to try and implement it yourself and see if it works?

[claymore666]

Currently I have only limited time, but I would try to look into it.

[jasperweiss]

The effort I put into securing my home assistant installation is pretty overkill. My domain points to a WireGuard interface address which only my devices can reach. On that IP a caddy webserver in docker serves a HTTPS reverse proxy to home assistant. Getting automatic Let's Encrypt certificates for a non-publicly routable IP has special challenges and HTTPS on top of WireGuard was mostly done to get rid of the browser warnings. Anyways..

I ordered a E32 gateway from BBQKees today but I'm just now finding out that there's absolutely no way to get even a basic level of security. Not even MQTT payload encryption using a pres-shared key. So it would seem that I would have to plug its ethernet cable into something like a raspberry pi which then routes the traffic securely. Are there any other options?

[proddy]

can you code? I'd love to add SSL (as this issue references).

[jasperweiss]

Yes perhaps I could work on adding that. I’m not too familiar with embedded development however and I haven’t received the board yet so I have nothing to test on just yet. I’m wondering if full blown TLS is the right solution to this problem however. You don’t really need all the cipher suites, negotiation steps and certificate chains when you’re just connecting 2 of your own devices together.

Another option is payload encryption for MQTT but it’s not part of the standard and support for it is lacking. It doesn’t help with the web portal either.

There do seem to be some options for using WireGuard on ESP32 which is just creating an additional network interface on the device which takes care of routing IP packets securely using just pre configured keypairs. You solve secrecy, integrity and authentication without needing any changes to MQTT or the webserver itself. It’s also a lot less computationally intensive so it’s better suited to embedded devices.

Essentially, you’d generate a set of WireGuard key pairs for the ESP32 board and your laptop and/or whichever device hosts homeassistant. You stick the keys on them and use the WireGuard IP address to route http and MQTT. Your browser, home assistant, etc are completely oblivious to it and work just the same. WireGuard was actually designed to be able to run on low power embedded devices and super computer clusters alike and you can essentially just slap it on the devices in your network and go about networking in your applications (plain, http, mqtt or whatever) as usual without ever thinking about network security again. You turn the whole thing into something that, security wise, is akin to applications talking on localhost except they can be in 2 different datacenters in different countries.

I’ll look into that first because it seems like the most elegant, secure and easy solution.

[jasperweiss]

So at first glance this seems to be really easy. See here.

I'd imagine you would open the web interface initially as usual, navigate to a section under security for wireguard where you stick some keys for other devices and optionally have the option to disable the webserver/MQTT on the eth/wlan ip addresses such that they are only reachable on the wireguard ip. Altough maybe the latter shouldn't really be necessary since the web ui and MQTT have username/password authentication and using the wireguard interface exclusively will prevent these credentials from being compromised anyway while having the webserver still reachable at the device's ip will allow you to get back in should you ever lose the wireguard keys.

Note that wireguard is stateless. There is no "connection" or complicated back and forth communication going on at any point as would be the case with TLS and it does not care about the ip address that packets originate from so roaming works naturally. It just encapsulates packets as they arrive at the interface and routes them using UDP, otherwise it is completely silent. It is invisible to any device that does not have a key since it does not reply to unauthenticated packets at all. Although not applicable to embedded devices, wireguard is also included in the mainline linux kernel and operates entirely from kernel space so the amount overhead is as low as it gets.

I feel like this suits the usecase of connecting embedded devices much better than TLS which is intended for clients to connect to arbitrary servers that may support different encryption suites, protocols and must prove their identity through complicated certificate hierarchies provided by trusted 3rd parties. Is this something you'd be interested in @proddy? I'd be happy to add it.

Edit: some work has already been done to get wireguard working on ESP Home https://github.com/esphome/feature-requests/issues/1444

[proddy]

Yes, absolutely. I agree that using WireGuard is a more elegant and less obtrusive solution than going the TLS route. I have WireGuard running in my proxmox ve so a little familiar with its setup. We can make a branch as soon as I cut v3.5 (next week I'm hoping) and work on that together

[jasperweiss]

As for TLS; It seems the library used for MQTT supports TLS. Isn't it just a matter of adding the frontend bits then?

[proddy]

As for TLS; It seems the library used for MQTT supports TLS. Isn't it just a matter of adding the frontend bits then?

probably, haven't looked into it

[proddy]

SSL was added to MQTT in 3.6.0.

[proddy]

Going to look into https://github.com/hoeken/PsychicHttp which is a replacement for ESPAsyncWebServer using the official espressif IDF, supporting SSL natively.

@MichaelDvP FYI. fun project

Will use the branch https://github.com/emsesp/EMS-ESP32/tree/https_36

[proddy]

I finished the port from AsyncWebServer to PsychicHttp. There are still some performance issues I need to fix as it's not behaving like it should. The good news is that the RAM has gone down now we're using AsyncTCP or AsyncWebServer. Also the code is cleaner. The bad news is that available heap has gone down by 10kb and I'm working with the library owner and espressif to see how to optimize this.

FYI @MichaelDvP

[proddy]

SSL/https is working too now, although its slow. I'm still not 100% happy with the IDF's web server. Overall it's slower and uses more heap memory than our highly optimized AsyncWS and AsyncTCP. So I'll keep this open until more work is done on the IDF library and it moves to using multi-thread in IDF 5.1.3

[proddy]

Also, just adding so I don't forget, the JWT is pretty useless without SSL. It's created from the username so if no one changes 'admin' it'll be the same token for everyone.

[jasperweiss]

SSL/https is working too now, although its slow.

I'm still not sure if SSL/TLS makes sense for small embedded devices. I'm not too familiar with ESP32 development but it seems WireGuard works well now . Would that be difficult too add?

[proddy]

we talked about wireguard a year ago, if you want to give it a shot. The SSL/TLS has been coded, although its very slow (as expected) but I will give you an extra security layer for those nervous folks.

[proddy]

2108