Feature request: secure WebSocket (wss)

[beyonlo]

Hello!

Congratulations for the great project.

I would like to know if you have intention to support secure WebSocket (use SSL over WebSocket) on the Microdot.

Thank you.

[miguelgrinberg]

SSL and WebSocket are completely independent features. I have the intention to investigate adding SSL support. I wasn't considering websocket, but it's not out of the question either. My concern is that with each addition the size of the package will continue to grow and the lower-end microprocessors will not be able to run anymore, so I'll have to think about doing this in a way that doesn't affect the smaller platforms.

[beyonlo]

SSL and WebSocket are completely independent features. I have the intention to investigate adding SSL support. I wasn't considering websocket, but it's not out of the question either. My concern is that with each addition the size of the package will continue to grow and the lower-end microprocessors will not be able to run anymore, so I'll have to think about doing this in a way that doesn't affect the smaller platforms.

Well, the microdot webserver/framework do not support HTTPS (SSL over HTTP) right? I think, even lower-end microcontrollers (like as ESP32-C3) would be great a secure connection over browser to configure/operate. So, if you add SSL on HTTP, the websocket can be the same way - I think that SSL is what will use more memory, right? Well, already exists a official module uwebsocket used by REPL. Here has a official uwebsocket module example (https://github.com/micropython/micropython/blob/master/extmod/webrepl/webrepl.py) but unfortunately do not use the uasyncio, but maybe that can help to no increase the microdot so much (using already exists uwebsocket module instead create your own). And already exists a HTTP Server official example running with SSL https://github.com/micropython/micropython/blob/master/examples/network/http_server_ssl.py but that not works with uasyncio too, unfortunately.

Well, my intention (my application) is to put in ESP32 works a secure HTTPS Server (to serve pages) + secure WebSocket Server, both running on uasyncio. I posted in MicroPython page, a time ago, a issue looking for a Simple example of uasyncio WebSocket Server (secure - with SSL) https://github.com/micropython/micropython/issues/8177

I think that most work to support ssl + asyncio / websockets was done, in many PRs, like this PR shows https://github.com/micropython/micropython/pull/5611

Thank you in advance!

[miguelgrinberg]

I believe uwebsocket does not work with uasyncio. But in general I agree, I'm open to add support for both SSL and WS.

[beyonlo]

I believe uwebsocket does not work with uasyncio. But in general I agree, I'm open to add support for both SSL and WS.

Hey @miguelgrinberg

That's will be great :)

I see that you have two microdot on/src/: microdot.py for people that want to use with thread microdot_asyncio.py to works with uasyncio/asyncio

Well, just as suggestion, maybe a option is have more options, so user can to choose what want: microdot_https.py Secure WebServer
microdot_https_asyncio.py Secure WebServer uasyncio/asyncio microdot_https_wss.py Secure WebServer + secure WebSocket Server microdot_https_wss_asyncio.py Secure WebServer + secure WebSocket Server uasyncio/asyncio

Particularly I liked so much uasyncio/asyncio :)

Going forward, with this feature in microdot is possible great applications. I can have the mobile app, desktop app and web app (microdot serving) all working with ESP32 using the same protocol, the websocket. That is great not just because is the same protocol, but is bidirectional protocol. Actually, using just http request on webserver, I need to refresh every 1-5s the page to have new data from my sensors. With websockets that polling is not more necessary. That was just a example, but I think that secure WebServer + secure WebSocket Server + uasyncio/asyncio will be an amazing feature to provide great applications/products.

Thank you so much!

[miguelgrinberg]

Update: WebSocket support is coming in the next release of Microdot. Will investigate SSL support after that.

[beyonlo]

Update: WebSocket support is coming in the next release of Microdot. Will investigate SSL support after that.

@miguelgrinberg That is a amazing news!!! I'm very interested!!

Actually I started to use this project https://github.com/marcidy/micropython-uasyncio-webexample to have a HTTP Server and a Websocket Server. My intention is to have just one HTML page with an entire large web application using javaScript (Jquery), exactly as that project works. This way is good because all processing will be on Client side (browser), not in the Microcontroller. As the my application has no access to the internet, is needed to put inside HTTP Server the JQuery lib do use on that unique page, exactly as that project works as well. Look here https://github.com/marcidy/micropython-uasyncio-webexample/tree/main/www

Will be possible to works in that way (just one page.html and put JQuery lib inside HTTP Server) on the Microdot as well?

Thank you very much.

[miguelgrinberg]

Will be possible to works in that way (just one page.html and put JQuery lib inside HTTP Server) on the Microdot as well?

You can already serve static files with Microdot. I have added an example of how to do this just a few days ago: https://github.com/miguelgrinberg/microdot/tree/main/examples/static.

And for websocket it will be just a normal route. This isn't finished code, but if you want to have a look, here is a WebSocket echo example: https://github.com/miguelgrinberg/microdot/blob/websocket/examples/websocket/echo_async.py

[beyonlo]

You can already serve static files with Microdot. I have added an example of how to do this just a few days ago: https://github.com/miguelgrinberg/microdot/tree/main/examples/static.

Excellent! That will solve the use case to serve images, JQuery lib and others kind off static files! :)

And for websocket it will be just a normal route. This isn't finished code, but if you want to have a look, here is a WebSocket echo example: https://github.com/miguelgrinberg/microdot/blob/websocket/examples/websocket/echo_async.py

Great, very clean with uasyncio! Congrats!

On the next Microdot release, do you have plans do provide together to the WebSocket Server feature a HTML page (index.html) with an WebSocket Client example running on the browser? Like as a start point (ready to go)!

As soon the next version is released I will start to use Microdot! :)

[miguelgrinberg]

an WebSocket Client example running on the browser?

Here is one from another project of mine: https://github.com/miguelgrinberg/flask-sock/blob/main/examples/templates/index.html. I did not test it, but I think it should directly work with the echo example I linked above.

[beyonlo]

an WebSocket Client example running on the browser?

Here is one from another project of mine: https://github.com/miguelgrinberg/flask-sock/blob/main/examples/templates/index.html. I did not test it, but I think it should directly work with the echo example I linked above.

Very good. I will test that, thanks :)

Will investigate SSL support after that.

To support SSL on HTTP Server and WebSocket Server will be amazing. I don't know if you know, but recently was created a ticket (micropython/micropython#8915) that has a relation with this feature on Microdot - maybe help.

[beyonlo]

Hi @miguelgrinberg

I tested the WebSocket branch using this WebSocket client example (https://github.com/miguelgrinberg/flask-sock/blob/main/examples/templates/index.html) and works very well!

However after I finished the test, I observed an error on the terminal. That error do not stopped the send/receive data from the browser (WebSocket Client) test, and I can't to reproduce that error anymore. I will paste here this test with the error, that maybe you can investigate if is really an error or not.

websocket_async.py example (a merge from hello_async.py and echo_async.py):

from microdot_asyncio import Microdot
from microdot_asyncio_websocket import websocket

app = Microdot()

htmldoc = '''<!doctype html>
<html>
  <head>
    <title>Flask-Sock Demo</title>
  </head>
  <body>
    <h1>Flask-Sock Demo</h1>
    <p>Type <b>close</b> to end the connection.</p>
    <div id="log"></div>
    <br>
    <form id="form">
      <label for="text">Input: </label>
      <input type="text" id="text" autofocus>
    </form>
    <script>
      const log = (text, color) => {
        document.getElementById('log').innerHTML += `<span style="color: ${color}">${text}</span><br>`;
      };

      const socket = new WebSocket('ws://' + location.host + '/echo');
      socket.addEventListener('message', ev => {
        log('<<< ' + ev.data, 'blue');
      });
      socket.addEventListener('close', ev => {
        log('<<< closed');
      });
      document.getElementById('form').onsubmit = ev => {
        ev.preventDefault();
        const textField = document.getElementById('text');
        log('>>> ' + textField.value, 'red');
        socket.send(textField.value);
        textField.value = '';
      };
    </script>
  </body>
</html>
'''

@app.route('/echo')
@websocket
async def echo(request, ws):
    c = 0
    while True:
        data = await ws.receive()
        print('Received data from client: {} - type is: {}'.format(data, type(data)))
        data = '{}_{}'.format(data, c)
        await ws.send(data)
        c += 1

@app.route('/')
async def hello(request):
    return htmldoc, 200, {'Content-Type': 'text/html'}

@app.route('/shutdown')
async def shutdown(request):
    request.app.shutdown()
    return 'The server is shutting down...'

app.run(debug=True)

Output terminal:

$ mpremote run websocket_async.py 
Starting async server on 0.0.0.0:5000...
GET / 200
Received data from client: s - type is: <class 'str'>
Received data from client: a - type is: <class 'str'>
Received data from client: asdas - type is: <class 'str'>
Received data from client: asdasd - type is: <class 'str'>
Received data from client: close - type is: <class 'str'>
Received data from client: freeee - type is: <class 'str'>
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcafef0>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 154, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Received data from client: asda - type is: <class 'str'>
Received data from client: asaaa - type is: <class 'str'>
Received data from client: a - type is: <class 'str'>
Received data from client: a - type is: <class 'str'>
Received data from client: a - type is: <class 'str'>
Received data from client: a - type is: <class 'str'>
Received data from client: aasdasd - type is: <class 'str'>
Received data from client: as - type is: <class 'str'>
Received data from client: das - type is: <class 'str'>
Received data from client: d - type is: <class 'str'>
Received data from client: asd - type is: <class 'str'>
Received data from client: as - type is: <class 'str'>
Received data from client: close - type is: <class 'str'>

Output Browser:

[beyonlo]

@miguelgrinberg I did tests with the example above (websocket_async.py) to check the compatibility with different browsers.

Works: Chrome, Chromium, Microsoft Edge and Opera. Do not works: Firefox (errors below).

I have not tested others browsers, just these above.

Terminal output for Firefox test:

$ mpremote run websocket_async.py 
Starting async server on 0.0.0.0:5000...
GET / 200
Traceback (most recent call last):
  File "microdot_asyncio.py", line 353, in dispatch_request
  File "microdot_asyncio.py", line 410, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 93, in wrapper
  File "microdot_asyncio_websocket.py", line 69, in websocket_upgrade
  File "microdot_asyncio_websocket.py", line 7, in handshake
  File "microdot_websocket.py", line 50, in _handshake_response
NameError: name 'abort' isn't defined
GET /favicon.ico 404
GET /echo 500

Firefox browser output:

Thank you.

[miguelgrinberg]

Thanks, this is useful feedback.

[beyonlo]

Hello @miguelgrinberg

I did some more tests.

1. More browsers tested: Chrome for Android and Safari for Iphone (IOS). Both works.
2. I'm using ESP32-S3 with MicroPython 1.19.1, and all tests above (in others reply) was done via WiFi, using STA mode. Now I tested the ESP32-S3 as AP mode, and sometimes (not always) show this errors:

Ps1: one kind of error here is the same found in the reply above (that as running as STA mode). Others errors are a bit different. Ps2: one time was needed to stop the Microdot and start it again because client (browser) was not capable anymore to open the IP and Port of the AP (http://192.168.4.1:5000). I do not restarted the WiFi on the ESP32-S3 (AP mode), just restarted the Microdot and and all back to works. I understand that Microdot is transparent of network, but is something that can be done to fix that?

Received data from client: S - type is: <class 'str'>
Received data from client: D - type is: <class 'str'>
Received data from client: D - type is: <class 'str'>
Received data from client: D - type is: <class 'str'>
Traceback (most recent call last):
  File "microdot_asyncio.py", line 353, in dispatch_request
  File "microdot_asyncio.py", line 410, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 95, in wrapper
  File "<stdin>", line 49, in echo
  File "microdot_asyncio_websocket.py", line 17, in receive
  File "microdot_asyncio_websocket.py", line 36, in _read_frame
  File "uasyncio/stream.py", line 1, in read
OSError: [Errno 104] ECONNRESET
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcc0400>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 139, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
GET / 200
GET / 200
GET /echo 200
GET /echo 200
Received data from client: Hhh - type is: <class 'str'>
Received data from client: JSON - type is: <class 'str'>
Traceback (most recent call last):
  File "microdot_asyncio.py", line 353, in dispatch_request
  File "microdot_asyncio.py", line 410, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 95, in wrapper
  File "<stdin>", line 49, in echo
  File "microdot_asyncio_websocket.py", line 17, in receive
  File "microdot_asyncio_websocket.py", line 36, in _read_frame
  File "uasyncio/stream.py", line 1, in read
OSError: [Errno 104] ECONNRESET
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcae620>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 139, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
GET / 200
Received data from client: F - type is: <class 'str'>
Received data from client: G - type is: <class 'str'>
GET / 200
Received data from client: H - type is: <class 'str'>
Received data from client: JSON - type is: <class 'str'>
Traceback (most recent call last):
  File "microdot_asyncio.py", line 353, in dispatch_request
  File "microdot_asyncio.py", line 410, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 95, in wrapper
  File "<stdin>", line 49, in echo
  File "microdot_asyncio_websocket.py", line 17, in receive
  File "microdot_asyncio_websocket.py", line 36, in _read_frame
  File "uasyncio/stream.py", line 1, in read
OSError: [Errno 104] ECONNRESET
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcbdbb0>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 139, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Traceback (most recent call last):
  File "microdot_asyncio.py", line 315, in handle_request
  File "microdot_asyncio.py", line 69, in create
  File "microdot_asyncio.py", line 110, in _safe_readline
  File "uasyncio/stream.py", line 1, in readline
OSError: [Errno 104] ECONNRESET
Traceback (most recent call last):
  File "microdot_asyncio.py", line 353, in dispatch_request
  File "microdot_asyncio.py", line 410, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 95, in wrapper
  File "<stdin>", line 49, in echo
  File "microdot_asyncio_websocket.py", line 17, in receive
  File "microdot_asyncio_websocket.py", line 36, in _read_frame
  File "uasyncio/stream.py", line 1, in read
OSError: [Errno 104] ECONNRESET
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcb4320>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 139, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcb68d0>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 139, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET

[miguelgrinberg]

Looks like all the errors are caused by not handling the ECONNRESET error. I did not notice this error in my tests, it might be that they are used by the ES32 implementation, which I did not test. I'll look into handling this error properly.

[Carglglz]

Hi @beyonlo @miguelgrinberg As a part of a ongoing development effort in #8968 to bring SSLContext to MicroPython, I've done some test with microdot and so far the _thread version works (I've done tests in UNIX and ESP32 ports), using python requests (with verify=False since I'm using self-signed certs) and curl. Here is the diff to microdot.py:

git diff --staged src/*
diff --git a/src/microdot.py b/src/microdot.py
index 5b9e77e..48149a1 100644
--- a/src/microdot.py
+++ b/src/microdot.py
@@ -51,6 +51,8 @@ except ImportError:
     except ImportError:  # pragma: no cover
         socket = None

+import ssl
+

 def urldecode(string):
     string = string.replace('+', ' ')
@@ -847,7 +849,7 @@ class Microdot():
         """
         raise HTTPException(status_code, reason)

-    def run(self, host='0.0.0.0', port=5000, debug=False):
+    def run(self, host='0.0.0.0', port=5000, debug=False, key=None, cert=None):
         """Start the web server. This function does not normally return, as
         the server enters an endless listening loop. The :func:`shutdown`
         function provides a method for terminating the server gracefully.
@@ -882,7 +884,11 @@ class Microdot():
         self.server = socket.socket()
         ai = socket.getaddrinfo(host, port)
         addr = ai[0][-1]
-
+        ctx = None
+        if key:
+            ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+            ctx.load_cert_chain(cert,
+                                keyfile=key)
         if self.debug:  # pragma: no cover
             print('Starting {mode} server on {host}:{port}...'.format(
                 mode=concurrency_mode, host=host, port=port))
@@ -893,12 +899,17 @@ class Microdot():
         while not self.shutdown_requested:
             try:
                 sock, addr = self.server.accept()
+                if key:
+                    ssl_sock = ctx.wrap_socket(sock, server_side=True)
             except OSError as exc:  # pragma: no cover
                 if exc.errno == errno.ECONNABORTED:
                     break
                 else:
                     raise
-            create_thread(self.handle_request, sock, addr)
+            if key:
+                create_thread(self.handle_request, ssl_sock, addr, ctx)
+            else:
+                create_thread(self.handle_request, sock, addr, ctx)

     def shutdown(self):
         """Request a server shutdown. The server will then exit its request
@@ -927,7 +938,7 @@ class Microdot():
                     f = 405
         return f

-    def handle_request(self, sock, addr):
+    def handle_request(self, sock, addr, ctx):
         if not hasattr(sock, 'readline'):  # pragma: no cover
             stream = sock.makefile("rwb")
         else:
@@ -955,6 +966,8 @@ class Microdot():
             print('{method} {path} {status_code}'.format(
                 method=req.method, path=req.path,
                 status_code=res.status_code))
+        if hasattr(ctx, 'reset'):
+            ctx.reset()

     def dispatch_request(self, req):
         if req:

And the hello_tls_context.py example I use for testing:

from microdot import Microdot

app = Microdot()

htmldoc = '''<!DOCTYPE html>
<html>
    <head>
        <title>Microdot Example Page</title>
    </head>
    <body>
        <div>
            <h1>Microdot Example Page</h1>
            <p>Hello from Microdot!</p>
            <p><a href="/shutdown">Click to shutdown the server</a></p>
        </div>
    </body>
</html>
'''

@app.route('/')
def hello(request):
    return htmldoc, 200, {'Content-Type': 'text/html'}

@app.route('/shutdown')
def shutdown(request):
    request.app.shutdown()
    return 'The server is shutting down...'

with open('ec-cakey.pem', 'rb') as keyb:
    key = keyb.read()

with open('ec-cacert.pem', 'rb') as certb:
    cert = certb.read()

app.run(debug=True, key=key, cert=cert)

Sadly I don't know when or even if my implementation of SSLContext will be merged.

I will try with microdot asyncio version too, although not sure when exactly will be that.

[miguelgrinberg]

@beyonlo I have put some fixes based on your reports:

• Firefox should work now
• the ECONNRESET error is (hopefully) handled properly
• an index.html file is now served in all the WebSocket examples when you connect to the root URL
• preliminary support for websocket connections in the test client (incomplete, more work is needed still)
• the websocket decorator was renamed to with_websocket for consistency

Let me know if things look better now.

[miguelgrinberg]

@Carglglz Thanks for doing this work! This is going to make my work a lot easier when I attempt to do this. Sounds like I might need to use the older (current) SSL implementation though, since it is unclear when or if yours is going to be merged, correct? I also want to see if it is possible to put the SSL support in a separate extension, as I'm doing with WebSocket and other features, so that it does not continue to make the main server larger.

[beyonlo]

Hello @miguelgrinberg

@beyonlo I have put some fixes based on your reports:

That's great!

• Firefox should work now

I confirm that it works!

• the ECONNRESET error is (hopefully) handled properly

Yes, that old errors do not show anymore, thank you. But now I did two other tests and show these errors:

1. Killing the Browser after connected.

Killing the Chrome on Android: Ps: the error happen every time that browser is killed.

Received data from client: D
Received data from client: D
Received data from client: S
Traceback (most recent call last):
  File "microdot_asyncio.py", line 353, in dispatch_request
  File "microdot_asyncio.py", line 410, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 95, in wrapper
  File "<stdin>", line 17, in echo
  File "microdot_asyncio_websocket.py", line 17, in receive
  File "microdot_asyncio_websocket.py", line 37, in _read_frame
  File "microdot_websocket.py", line 63, in _parse_frame_header
IndexError: bytes index out of range
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fccb1c0>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 145, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcca9a0>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 145, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Received data from client: sfsd
Received data from client: f

Killing the Chromium on Ubuntu: Ps: the error happen just some times.

Received data from client: sd
Received data from client: a
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcc2450>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 145, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Received data from client: ds
Received data from client: asdasd
Received data from client: a

Killing the Firefox on Ubuntu do not show errors

2. Happened a non-intentional lost network, and it back a few seconds after, and show this error: Ps: I tried to force network lost connection to reproduce the problem, but not success.

Received data from client: as
Received data from client: d
Received data from client: asd
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcc9f20>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 145, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Received data from client: ada

• an index.html file is now served in all the WebSocket examples when you connect to the root URL

Excellent!

• preliminary support for websocket connections in the test client (incomplete, more work is needed still)

I never did/used unittests, but I will to study/check how it works and start to use the test_microdot_websocket.py as well in my tests. I think need just to instance the TestMicrodotWebSocket class and call the test_websocket_echo method, right? I checked that import unittest do not is builtin by default on MicroPython, but I see that there is the unittest for MicroPython.

• the websocket decorator was renamed to with_websocket for consistency

All right!

Additional comments:

1. I just noticed that this new version are not showing anymore IP:PORT that Microdot are running - show nothing. In the last version was showing something like as Running at 0.0.0.0:5000.
2. All tests above was done using the new echo_async.py on the ESP32-S3 running MicroPython 1.19.1. The ESP32-S3 was running as AP mode and my Notebook (Ubuntu) and my smartphone (Android) as STA mode, connected to ESP32-S3.

   ESP32-S3 - 192.168.4.1
   Notebook - 192.168.4.2
   Smartphone - 192.168.4.3

[miguelgrinberg]

@beyonlo I added fixes for the new errors. Thanks!

[raven703]

Confirm, that ECONNRESET has been fixed ! I`m using server side events in my ESP32 project and it was spamming in local each time browser reloads page forcing Microdot to stop serve pages. Now everything is fine, thank you :)

[beyonlo]

@miguelgrinberg I confirm that is working without ECONNRESET errors.

Thank you!

[beyonlo]

Hi @miguelgrinberg

Could you please to provide a simple example a bit different than echo_async.py using asyncio as well? I mean, where the WebSocket Server is capable to send data to the WebSocket Client without the WebSocket Client send a request (like as the echo example), and where the WebSocket Server is capable as well to select (choose) for what WebSocket Clients to send the data? Just a example as start point.

I would like to have many WebSockets Clients (from browsers and from Desktop/Mobile), simultaneously connected do the WebSocket Server and the clients will be stayed connected. So the WebSocket Server can receive/send data from/to all them, but for some critical data (like as alarms/events), for example, the WebSocket Server need to send the data just for some specific WebSocket Clients.

Thank you in advance!

[miguelgrinberg]

@beyonlo I don't have any examples, but all you need to do is store the ws objects for all your clients in some sort of data structure (maybe a dict), so that you can access them when you need to send something to one or more clients. The route can take care of receiving data from clients, but for sending you can send from anywhere, as long as you have access to the ws object.

[beyonlo]

@beyonlo I don't have any examples, but all you need to do is store the ws objects for all your clients in some sort of data structure (maybe a dict), so that you can access them when you need to send something to one or more clients. The route can take care of receiving data from clients, but for sending you can send from anywhere, as long as you have access to the ws object.

@miguelgrinberg with this explanation now I understand how to do it - I will to try, thanks!

In my tests I found more errors. I don't know when that errors happened and I was no capable to reproduce. Note that the second error (ENOTCONN) is different from all others errors reported before.

Received data: s
Send data: s -> 9
1070296480 1070296736
Received data: Vvvg
Send data: Vvvg -> 1
1070295920 1070296048
Traceback (most recent call last):
  File "microdot_asyncio.py", line 353, in dispatch_request
  File "microdot_asyncio.py", line 410, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 101, in wrapper
  File "microdot_asyncio_websocket.py", line 33, in close
  File "microdot_asyncio_websocket.py", line 28, in send
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcaed40>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 140, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 128] ENOTCONN

GET / 200
GET /static/jquery-ui.min.css 200
GET /static/jquery-3.6.0.min.js 200
GET /static/jquery-ui-1.13.2.min.js 200
---------------------
<Request object at 3fcb79e0> <WebSocket object at 3fcb7aa0>
<class 'Request'> <class 'WebSocket'>
1070299616 1070299808
---------------------
1070299616 1070299808
Received data: asdsa
Send data: asdsa -> 0
1070299616 1070299808
Received data: asd

Edit:

Happened one more time:

Received data: free
Send data: free -> 76
1070301872 1070302064
Traceback (most recent call last):
  File "microdot_asyncio.py", line 353, in dispatch_request
  File "microdot_asyncio.py", line 410, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 101, in wrapper
  File "microdot_asyncio_websocket.py", line 33, in close
  File "microdot_asyncio_websocket.py", line 28, in send
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 104] ECONNRESET
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3fcac1d0>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 261, in serve
  File "microdot_asyncio.py", line 321, in handle_request
  File "microdot_asyncio.py", line 140, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in drain
OSError: [Errno 128] ENOTCONN

[miguelgrinberg]

@beyonlo This is useful, thanks. I think I've fixed these new errors now.

[beyonlo]

@miguelgrinberg Errors do not happen anymore, thanks!

Do you know if is possible to limit on the WebSocket Server how many WebSockets clients can to connect?

[beyonlo]

Do you know if is possible to limit on the WebSocket Server how many WebSockets clients can to connect?

I did that just counting the ws objects appended on the dict, and after reach more than the max count, I just do ws.close() to end the WebSocket connection. I don't know if this approach is the best way to do that, so suggestions are welcome! :)

Thank you!

[miguelgrinberg]

Yes, that is a reasonable solution.

[miguelgrinberg]

@Carglglz Hey, I'm looking at your patch above and have a question. Why is it necessary to reset the SSLContext object with each request in your proposed implementation? This isn't in CPython's SSLContext, as far as I can see. Thanks!

[Carglglz]

Why is it necessary to reset the SSLContext object with each request in your proposed implementation? This isn't in CPython's SSLContext, as far as I can see. Thanks!

@miguelgrinberg , I've made some comments about this in #8968

I've just added SSLContext.reset() method and a http_server_ssl_context.py example. This allows to "reuse" the SSLContext. I guess this is not CPython compliant but I'm not sure how to balance between memory allocation and context reusability. Right now what context does (or I think it does) is:

Context initiation --> memory allocation --> load cert/key/cadata/conf Context wrap socket --> creates a ssl socket that use the allocated memory from context SSLSocket is closed and frees the allocated memory So to reuse the context, it needs to be reinitiated to allocate memory again, or another option would be to stop the SSLSocket from freeing the allocated memory (but I'm not sure this is the way to go... 🤔 )

TLDR: there is a tradeoff between memory and context reusability, to sum up, you increase memory usage and gain context reusability without any additional method or you save memory at expense of adding a reset method.

In CPython since there is memory to spare the choice is obvious, but for an embedded system I thought saving memory was the better choice. (There may be a better method I've not thought about, but for now that's what I got 🤷🏼‍♂️)

That's why I added if hasattr(ctx, 'reset') so it will work both with CPython and MicroPython. 👍🏼

[miguelgrinberg]

@Carglglz unless I'm missing something, your solution is flawed. You are assuming that there is going to be only one outstanding request, which is incorrect. If that was the case there wouldn't be a need to put the request handler in its own thread. I did not look at your implementation, but the SSLContext object should be able to wrap multiple sockets. Resetting the ssl context at random times just because a request handler completed seems strange at best, and a source of race conditions at worst.

[Carglglz]

@Carglglz unless I'm missing something, your solution is flawed. You are assuming that there is going to be only one outstanding request, which is incorrect.

@miguelgrinberg this is unfortunately the case at least in current ESP32 port, there is only enough memory to have one SSLSocket at a time. (I do have a custom ESP32 implementation where I can have up to two SSLSockets but I haven't tested doing multiple requests at a time there.)

If that was the case there wouldn't be a need to put the request handler in its own thread.

True, but the _thread version is the only version I could test (or is there any other non-threaded version I missed?)

SSLContext object should be able to wrap multiple sockets. Resetting the ssl context at random times just because a request handler completed seems strange at best, and a source of race conditions at worst.

This is also true if you have enough memory to spare and have multiple SSLSockets, so SSLContext could have the key/cert/cadata/conf allocation and when wrapping a socket the SSLSocket will get a copy of that instead of using the allocated one. This will allow to reuse the context and wrap multiple sockets at expense of memory.

Maybe in ESP32 with PSRAM and a custom allocation method. That will be the endgame actually.

The point is SSLContext is coming to MicroPython (I don't know exactly when) and it is intended to be CPython compliant. So whatever work you do to implement SSL support in microdot using CPython, should be compatible with MicroPython too (although maybe with some caveats due to resource constraints as you can see.)

[miguelgrinberg]

this is unfortunately the case at least in current ESP32 port, there is only enough memory to have one SSLSocket at a time.

@Carglglz Ah, okay, so that was the catch. I still think the design is flawed though, because there are platforms in which multiple SSL sockets are possible. The memory cleanup should be done by socket when it is closed and not globally for the entire context.

So whatever work you do to implement SSL support in microdot using CPython, should be compatible with MicroPython too

Yes. I have created a FakeSSLContext class that implements a wrap_socket method using ssl.wrap_socket underneath, and that seems to work.

[miguelgrinberg]

@beyonlo @Carglglz TLS/SSL support is now committed to the main branch. Couple of examples here: https://github.com/miguelgrinberg/microdot/tree/main/examples/tls.

Would appreciate it if you test it and provide feedback. Note that the async support cannot currently be used with MicroPython, since uasyncio does not have the necessary support.

[Carglglz]

@miguelgrinberg I've just tested and it works both in CPython and current MicroPython UNIX port 👍🏼 . This is what I meant, once SSLContext is implemented in MicroPython changes to microdot_ssl.py should be minimal or even non-existent (apart from code deprecation as you have already indicated)

[beyonlo]

@miguelgrinberg I tested on the ESP32-S3 with MicroPython 1.19.1 (same of all others tests) but do not works. When I try to access via browser, Microdot stop and show errors. Follow the details:

$ mpremote run hello_tls.py 
Starting sync server on 0.0.0.0:4443...
Traceback (most recent call last):
  File "<stdin>", line 36, in <module>
  File "microdot.py", line 914, in run
  File "microdot_ssl.py", line 46, in accept
OSError: (-30592, 'MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE')

$ mpremote ls
ls :
         139 boot.py
        1493 cert.der
        2375 key.der
       37277 microdot.py
       16463 microdot_asyncio.py
        3780 microdot_asyncio_websocket.py
        2362 microdot_ssl.py
        5935 microdot_websocket.py

Chromium output:

Firefox output:

[Carglglz]

@beyonlo This is expected as you are using self signed certificates, to make it work with your web browser you need to add the self signed certificate to your system trusted certificates. Look for adding self signed certificate to chrome or whatever browser you are using.

[beyonlo]

@Carglglz yes, you are correct. I added that exception and works - screenshot below. But the Microdot do not should to stop in that exception, I believe.

@miguelgrinberg it is already possible as well to use Secure WebSocket (wss), os just the HTTPS for now?

[Carglglz]

@miguelgrinberg based on your feedback I've modified my implementation of SSLContext, and now the context is automatically regenerated after the handshake (so it can wrap new sockets) and also note that in CPython context.wrap_socket can wrap connected or not connected sockets, which I tried to support as well, and in UNIX port so far so good, but I'm not sure if the extra code worth it in an embedded platform. With these modifications, tests and your hello_tls.py example are working nice.

I have to try them in ESP32 port now...

[EDIT] Works on ESP32 port too but the CPython context.wrap_socket feature on not connected sockets its limited and unstable, I cannot get more than 4 in a row... maybe its better to enforce connect first since CPython already supports this.

[beyonlo]

@miguelgrinberg this is unfortunately the case at least in current ESP32 port, there is only enough memory to have one SSLSocket at a time. (I do have a custom ESP32 implementation where I can have up to two SSLSockets but I haven't tested doing multiple requests at a time there.)

@Carglglz Is true that will be not possible to have more than 1 Secure WebSocket using the ESP32-S3?

Actually I'm using the Microdot WebSocket Server, with an intention to have many WebSocket Clients (Browsers and Desktop applications), all them simultaneously connected and communicating with the Microdot WebSocket Server running on the ESP32-S3 - that is working.

My intention is after that HTTPS/wss works, I can just upgrade to support secure Websockets in that scenario of multi WebSockets.

I need to use the ESP32-S3 without PSRAM for special needs, but even the ESP32-S3 with just internal RAM, it has a large Free memory (~170KB):

>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 168704, used: 1584, free: 167120
 No. of 1-blocks: 27, 2-blocks: 8, max blk sz: 18, max free sz: 10432
>>> 

I checked that for each WebSocket Client connected (persistent connection) to the Microdot WebSocket Server, is allocated around ~2.2KB of RAM. I have 8 WebSockets clients connected to Microdot WebSocket Server using total of ~18KB RAM (just for that 8 persistent connections).

Sorry, but when you say that the ESP32 has only enough memory to have one secure Webocket (SSLSocket) at a time, I imagine that one Secure WebSocket connection will use more than 50/100KB. Is that realistic?

Thank you in advance!

[miguelgrinberg]

@beyonlo Correct, this should not stop Microdot. If it does, then that is an exception that I'm not catching, I'll look into that. Regarding WebSocket, I think it should work, but I have not tested wss yet.

@Carglglz I did see that you cannot wrap the main socket in MicroPython. This fake SSL context class that I built allows you to do it, but it is handled internally by delaying the actual call to wrap_socket until after a client socket is available.

[Carglglz]

@beyonlo SSL use HEAP memory not RAM memory from MicroPython, there is some work to be able to support usage of RAM memory too so it can support multiple SSL sockets, check #8940

@miguelgrinberg yes, I've tried to do the same with mixed results in ESP32 port, for server socket it does work at expense of increasing handshake time, and for client sockets I cannot get more than 4 handshakes in a row...

[EDIT] I've modified my tests and it seems to be working on ESP32, reusing the context instead of creating new instances does the trick in client side 👍🏼.

For server side it's still better to enforce connect first since CPython already supports this, and it saves a considerable amount of memory since it does not create additional objects. Also it's far more stable.

[beyonlo]

@beyonlo SSL use HEAP memory not RAM memory from MicroPython, there is some work to be able to support usage of RAM memory too so it can support multiple SSL sockets, check #8940

@Carglglz understood! Thank you for the explanation.

[Carglglz]

Update: WebSocket support is coming in the next release of Microdot. Will investigate SSL support after that.

@miguelgrinberg @beyonlo I did a test with WSS and it does work 👍🏼 . It was not too difficult either.

Just modify index.html to use wss

diff --git a/index.html b/../../_local_tests/index.html
index 6e240d5..51d6b86 100644
--- a/index.html
+++ b/../../_local_tests/index.html
@@ -16,7 +16,7 @@
         document.getElementById('log').innerHTML += `<span style="color: ${color}">${text}</span><br>`;
       };

-      const socket = new WebSocket('ws://' + location.host + '/echo');
+      const socket = new WebSocket('wss://' + location.host + '/echo');
       socket.addEventListener('message', ev => {
         log('<<< ' + ev.data, 'blue');
       });

And in MicroPython a fake SSLSocket class with missing send, recv methods and readline that can accept 1 arg.

class sslsocket_class:
    def __init__(self, sock):
        self.sock = sock

    def write(self, buf):
        return self.sock.write(buf)

    def read(self, size):
        return self.sock.read(size)

    def readline(self, maxbuff=None):
        if not maxbuff:
            return self.sock.readline()
        else:
            return self.sock.readline()[:maxbuff]

    def readinto(self, *args):
        return self.sock.readinto(*args)

    def setblocking(self, flag):
        return self.sock.setblocking(flag)

    def makefile(self, *args, **kwargs):
        return self.sock.makefile(*args, **kwargs)

    def send(self, buf):
        return self.sock.write(buf)

    def recv(self, buf):
        return self.sock.read(buf)

    def __del__(self):
        self.sock.__del__()

    def getpeercert(self, *args):
        return self.sock.getpeercert(*args)

    def cipher(self):
        return self.sock.cipher()

    def close(self):
        self.sock.close()

Finally wss_echo.py

from microdot import Microdot, send_file
from microdot_websocket import with_websocket
from microdot_ssl import create_ssl_context
import sys

app = Microdot()

@app.route('/')
def index(request):
    return send_file('index.html')

@app.route('/echo')
@with_websocket
def echo(request, ws):
    while True:
        data = ws.receive()
        ws.send(data)

ext = 'der' if sys.implementation.name == 'micropython' else 'pem'
sslctx = create_ssl_context('cert.' + ext, 'key.' + ext)
app.run(port=4443, debug=True, ssl=sslctx)

[beyonlo]

@Carglglz That's great! Maybe @miguelgrinberg can to generate a ready to go wss example and put in the examples directory to tests. I can test it, like as I did with the HTTPS example, but in my case, I need that HTTPS and wss works with MicroPython uasyncio support :) I think that will be supported soon on the uasyncio!

[miguelgrinberg]

Thanks. Pushed a fix for wss and also an example.

[beyonlo]

@miguelgrinberg I tested the new echo_tls.py example on the ESP32-S3 and it works, but some errors happened until I accept the self signed certificates. Follow the details:

$ mpremote run echo_tls.py 
Starting sync server on 0.0.0.0:4443...
Traceback (most recent call last):
  File "microdot.py", line 914, in run
  File "microdot_ssl.py", line 46, in accept
OSError: (-30976, 'MBEDTLS_ERR_SSL_BAD_HS_CLIENT_HELLO')
Traceback (most recent call last):
  File "microdot.py", line 914, in run
  File "microdot_ssl.py", line 46, in accept
OSError: (-30592, 'MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE')
Traceback (most recent call last):
  File "microdot.py", line 914, in run
  File "microdot_ssl.py", line 46, in accept
OSError: (-30592, 'MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE')
GET / 200

[miguelgrinberg]

@beyonlo these errors did not stop the application, so they are not a cause for concern. Given that it is impossible for me to know all the possible error codes, and when and if it is safe to suppress the error, the option that I prefer is to log the errors and continue, which is what I'm doing here.

[beyonlo]

@miguelgrinberg I agree! To log the errors and continue is the really the best option, so if something more happen, there is a error log to analyse!