search

adafruit / circuitpython

CircuitPython - a Python implementation for teaching coding with microcontrollers
https://circuitpython.org
Other
4.09k stars 1.21k forks source link

HTTPS Server works on raspberrypi but not on espressif #8268

Closed ams1 closed 8 months ago

ams1 commented 1 year ago

Hi,

Thank you for this AWESOME initiative!

Any plan to have an example for SSL http server?

michalpokusa commented 1 year ago

Hi, thanks for kind word.

Unfortunately right now implementing SSL is problematic, due to the amount of compute power necessary, it is very slow and in my testing does not work at all on some microcontrollers.

Related issue on CircuitPython itself: https://github.com/adafruit/circuitpython/issues/7657

Attempt to make a HTTPS server: https://github.com/ide/circuitpython-https-server

If you want to play with it, I would appreciate you sharing your results.

But if you really need a HTTPS, one way is to use e.g. nginx on Raspberry Pi (Zero/4), but at this point you can simply host a website on it too.

zap8600 commented 1 year ago

I've tried using the example on CircuitPython 8.2.0, using both version 4.2.0 and 2.3.0, on an Adafruit Feather ESP32-S2. The error I get is the same on both HTTP server versions. The error below is from my attempt to use version 2.3.0, modified to print the socket type on server start. Here is the error I'm getting.

Traceback (most recent call last):
  File "code.py", line 79, in <module>
  File "code.py", line 49, in main
  File "/lib/adafruit_httpserver/server.py", line 137, in poll
RuntimeError: Invalid socket for TLS

Here is the line the error is happening at.

            conn, client_address = self._sock.accept()

Hope this helps!

anecdata commented 1 year ago

What are you using for a client? curl -v might yield some clues.

michalpokusa commented 1 year ago

During my testing I was getting the exact same error. I was testing using curl, VSCode extension Thunder Client, a script in Python with requests, Google Chrome and even other microcontroller (ESP32-S2 TFT) with adafruit_requests.

I believe it is something in the CP itself. The interesting thing is that adafruit_requests as far as I know can make requests to HTTPS endpoints, so the capability is there.

zap8600 commented 1 year ago

What are you using for a client? curl -v might yield some clues.

I'm using curl. curl will stall, then throw an error once the microcontroller resets.

anecdata commented 1 year ago

The curl switch -v gives detail about the SSL transaction, I thought it might give a clue where in the process it breaks.

zap8600 commented 1 year ago

The curl switch -v gives detail about the SSL transaction, I thought it might give a clue where in the process it breaks.

I haven't looked much into it, but I think it only has to do with CP. Both curl and the browser don't get anything back from the microcontroller. They'll throw a connection reset error when the microcontroller resets, though.

anecdata commented 1 year ago

I loaded up a Pico W and a QT Py S2 and a QT Py S3 (all on CP 8.2.0, all freshly reset).

It works on Pico W:

% curl -v --insecure https://192.168.6.198
*   Trying 192.168.6.198:443...
* Connected to 192.168.6.198 (192.168.6.198) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=picow.local
*  start date: Feb 28 05:42:56 2023 GMT
*  expire date: Feb 28 05:42:56 2024 GMT
*  issuer: CN=picow.local
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 192.168.6.198
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Length: 117
< Content-Type: text/html
< Connection: close
< 
<!DOCTYPE html>
<title>Hello world &middot; Raspberry Pi Pico W</title>
<p>Hello world from Raspberry Pi Pico W!</p>
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):

It breaks early in the TLS handshake on S2 and S3:

% curl -v --insecure https://192.168.6.210
*   Trying 192.168.6.210:443...
* Connected to 192.168.6.210 (192.168.6.210) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
* Closing connection 0
curl: (35) Recv failure: Connection reset by peer

Same behavior with CP 8.2.2 on S2.

Probably some difference in common-hal, or esp-idf. Perhaps we should move this issue to the circuitpython repo.

anecdata commented 1 year ago

code.py passes in server_side=True in line 74: return self._ssl_context.wrap_socket(socket, server_side=True) https://docs.circuitpython.org/en/latest/shared-bindings/ssl/index.html#ssl.SSLContext.wrap_socket espressif does nothing with it: https://github.com/adafruit/circuitpython/blob/6b36bf59e30a4d3d93ec9df1f92ae8adc5fe2094/ports/espressif/common-hal/ssl/SSLContext.c#L40 compared to raspberrypi: https://github.com/adafruit/circuitpython/blob/6b36bf59e30a4d3d93ec9df1f92ae8adc5fe2094/ports/raspberrypi/common-hal/ssl/SSLSocket.c#L166

zap8600 commented 1 year ago

So we just need to modify the espressif port to check for the argument?

anecdata commented 1 year ago

There's presumably some lwip logic needed, predicated on server_side=True. But it is odd that the exception is Invalid socket for TLS, which I think indicates the socket isn't SOCK_STREAM, though the library sets it when it creates the socket to listen to.

dhalbert commented 1 year ago

We could backport this to 8.2.x if it gets figured out soon.

michalpokusa commented 11 months ago

Is there any progress on fixing/updating ESPs to allow HTTPS? I would like to make an update to adafruit_httpserver that allows HTTPS, but currently for all I know only RPi Pico W supports what is needed for it to work.

zap8600 commented 10 months ago

Why don't we just add

    ret = mbedtls_ssl_config_defaults(&o->conf,
        server_side ? MBEDTLS_SSL_IS_SERVER : MBEDTLS_SSL_IS_CLIENT,
        MBEDTLS_SSL_TRANSPORT_STREAM,
        MBEDTLS_SSL_PRESET_DEFAULT);
    if (ret != 0) {
        goto cleanup;
    }

to common_hal_ssl_sslcontext_wrap_socket? Is there more that's supposed to be done? I'll make a fork and test this.

tannewt commented 8 months ago

@michalpokusa or @anecdata did you try @zap8600's suggestion?

jepler commented 8 months ago

I don't think calling mbedtls APIs here will work, the espressif SSLContext is using ep-idf calls like esp_tls_init.

It's tempted to think we could/should use mbedtls API calls (not esp-idf calls) for both espressif & raspberrypi rp2 (I think this is what micropython does!) but that would be a big change.

anecdata commented 8 months ago

Does anyone have working test code for HTTPS Server for espressif with this PR? I'm getting RuntimeError: Invalid socket for TLS using code from https://github.com/ide/circuitpython-https-server updated for latest adafruit_httpserver. (I may be doing something wrong)

i.e., how do we know this is completed?

michalpokusa commented 8 months ago

Does anyone have working test code for HTTPS Server for espressif with this PR? I'm getting RuntimeError: Invalid socket for TLS using code from https://github.com/ide/circuitpython-https-server updated for latest adafruit_httpserver. (I may be doing something wrong)

i.e., how do we know this is completed?

I also started trying HTTPS with adafruit_httpserver on ESP32-S2 TFT Feather. I get exactly the same error, specifically it gets raised by Server._sock.accept() in Server.poll().

anecdata commented 8 months ago

client side (essentially same output for ESP32-S3 and Pico W:

% curl -v --insecure https://192.168.6.243
*   Trying 192.168.6.243:443...
* Connected to 192.168.6.243 (192.168.6.243) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
* Closing connection
curl: (35) Recv failure: Connection reset by peer
jepler commented 8 months ago

@anecdata please re-test with #8932, which fixes the "Invalid Socket for TLS" message. I tested on esp32-s3 (matrixportal s3) with https://github.com/ide/circuitpython-https-server/pull/2

(however if it's not working on pico w either there's something else going on as well. That port did already set the socket type on an accepted socket, as far as I could tell from reading the source code)

anecdata commented 8 months ago

@jepler works for me now on espressif, and it's plenty fast (QT Py S3)... Thanks!

michalpokusa commented 8 months ago

@jepler I also tested on my MatrixPortal S3, works very fast, I also tested with Websockets and wss://.

But I was unable to make it work with ESP32-S2 Feather TFT, I can start the server but when trying to connect I get an empty MemoryError:. Seems strange considering that both have 2MB of RAM, although MatrixPortal has SRAM and Feather has PSRAM, would that make any difference?

@anecdata Could you maybe test on the Feather if you have one on hand?

EDIT1: MemoryError: is present even with .mpy version of the adafruit_httpserver

anecdata commented 8 months ago

I get the same on QT Py ESP32-S2 (N4R2), no dual http/https servers (just https):

server:

Traceback (most recent call last):
  File "adafruit_httpserver/server.py", line 404, in poll
MemoryError: 
Traceback (most recent call last):
  File "code.py", line 61, in <module>
  File "adafruit_httpserver/server.py", line 450, in poll
  File "adafruit_httpserver/server.py", line 404, in poll
MemoryError:

client:

$ curl -iL -v --insecure https://192.168.6.180?HTTPS
*   Trying 192.168.6.180:443...
* Connected to 192.168.6.180 (192.168.6.180) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* Recv failure: Operation timed out
* LibreSSL/3.3.6: error:02FFF03C:system library:func(4095):Operation timed out
* Closing connection 0
curl: (35) Recv failure: Operation timed out

edit: yes, EESP32-S3 has more internal SRAM than ESP32-S2, but maybe there could be build differences or something else to account for it? I suspect there's some difference of SRAM available to esp-idf.

michalpokusa commented 8 months ago

If RPicoW with 264kB can handle HTTPS, maybe there is something else here that needs to be fixed? It is definitely better but still not completely working.

Would it be worth re-opening this issue?

anecdata commented 8 months ago

I think it needs an issue, either re-open or new. Some difference in S2 vs. S3, and also "https:// PicoW" breaks with latest everything (unless you've gotten it to work?)

iirc S2 has 320KB SRAM, and S3 has 540KB 512KB SRAM, with various claims to that memory from esp-idf/rtos/etc, and maybe some things from CP core too. I don't fully understand the new CP 9 memory model.

michalpokusa commented 8 months ago

I double checked and you are right, RPicoW now has the same problem as Feather, which is MemoryError:.

I assumed that considering that only the espressif port changed, code would still work on raspberry.

michalpokusa commented 8 months ago

After updating to CP 9.0.0-beta.1 I get a different error: image

This does not mean that the MemoryError is gone, as it might appear later, but something else changed that unables even ESP32-S3 boards like MatrixPortal S3 to start a HTTPS server.

anecdata commented 8 months ago

Pico W HTTPS Server seems to have broken with the MemoryError somewhere between 8.0.0 and 8.1.0.

tannewt commented 8 months ago

iirc S2 has 320KB SRAM, and S3 has ~540KB~ 512KB SRAM, with various claims to that memory from esp-idf/rtos/etc, and maybe some things from CP core too. I don't fully understand the new CP 9 memory model.

CP will try to allocate to PSRAM first now in beta.1+. (Not exactly sure when I changed it.) The IDF can allocate to PSRAM now too by default but needs to request it explicitly. Without an explicit request, then internal memory will be used first.