Yielding data from an async task to a chunked response

[adrianblakey]

How do I yield data to a chunked response from a async routine?

The code reads the adc like this:

async def read_adc(pin: microcontroller.Pin):
    with analogio.AnalogIn(pin) as adc:
        while True:
            local_value = (adc.value * 3.3) / 65536
            await asyncio.sleep(0.1) 

How can the local_value be yield'ed in a chunked response?

[michalpokusa]

From the code that you provided it seems that this is a continuous stream of data. The way that a ChunkedResposne is intended to use is mainly when returning a large response to split it, so that you do not need to store the whole response in memory at any moment. If this function never returns, the chunked response will never be "finished".

Also, there is no oficial support for async at the moment, although it stil might be possible to use that.

Depending on what you are trying to achieve, you might want to try websockets, MQTT or even saving values to list and returning them on request.

Keep in mind, that CircuitPython is single threaded, and the adafruit_httpserver can handle one request as the time, and only after it is finished, it will handle the next one.

[adrianblakey]

ty

Please could you provide some links to an example that uses websockets?

[michalpokusa]

ty

Please could you provide some links to an example that uses websockets?

I have not worked with websockets in CircuitPython, but i know Neradoc has a library for that, you will find some examples there. https://github.com/Neradoc/websockets-for-circuitpython

[Neradoc]

Note that we don't have a websockets server implementation that I know of, only client.

[michalpokusa]

Note that we don't have a websockets server implementation that I know of, only client.

Oh, i didn't know that. Thanks for clearing that out.

In that case MQTT might be the way to go.

[adrianblakey]

What's the simplest, easiest and fastest way to stream data from the Pi Pico W to a client?

It must be a common use case to read data from some sensor or device, possibly cache it, and continuously graph it in a browser.

• Websockets isn't an option - no websockets on the http server - what does it take to get it implemented? Looks like it's possible in micropython https://www.donskytech.com/using-websocket-in-micropython-a-practical-example/ - time to switch?
• A hack might be to write to the local filesystem, poll this file from the browser and truncate/replace/switch it as it fills up.
• Then there is Redis - there's this for micropython https://micropython-redis.readthedocs.io/en/latest/ might not be too hard to port it to socketpools.
• MQTT looks like overkill for a simple point to point solution ...

btw - am I right in assuming that circuit python "workflows" are really circuit python kernel implementations for reading/writing data to the device? Took me forever to have the aha.

I believe there is a defect in assignment to wifi.radio.hostname. If it's reused in the same execution new values are merged and do not replace the existing value. E.g. wifi.radio.hostname = 'some-long-foo' then later wifi.radio.hostname = 'short-name' the resulting value will be: short-namefoo

I discovered this in code to uniqueify my hostname, be setting the hostname to some random string, running a getaddrinfo to find the hostname I want to use. Resolving clashes with the desired name by suffixing the name and looking again until there's no match, stop station, set the hostname to the unique value and connect again.

[michalpokusa]

Regarding the hostname problem, I tested your scenario on ESP32-S2 and everything works as expected, I do not have a Pico W with me at the moment so can't test that. If you provide a exact snippet that you used I can you it to check again. Maybe it is a Pico specific thing.

When it comes to your project, the most appropriate solution highly depends on:

• how many Pico W will send data
• do you need historic data, like from 5minutes/2 hours ago, or only after a client connects
• do you need it in real time, <1s delay, or can it be more e.g. 10s
• do you want to store the data, or only use it at runtime

[adrianblakey]

I really appreciate your help :-) tysm

I checked in the code here: https://github.com/adrianblakey/slot-car-data-logger

I hope the README.md answers the questions - if not please tell me

btw I am running: Adafruit CircuitPython 8.2.0-rc.1 on 2023-06-27; Raspberry Pi Pico W with rp2040

[michalpokusa]

So, after reading the README.md and some research I found the SSE (Server-Sent Events), which seem to suit your needs. Today I managed to write experimental logic for supporting it in this repo branch, I believe it would be a good addition to lib. There are some limitations, like max number of connected client, which seems to be 1 because of the limit on TCP sockets etc.

This should work for your application, the usage is limited, but Websockets probably would have the same problems for the same reason, but we have to remember we operate on microcontrollers, not fully-featured computers.

If you happen to need help implementing SSE in your project feel free to contact me, or ask on Adafruit's Discord Server.

[adrianblakey]

I tried the example - a very cool solution, ty.

The first GET request to /client downloads the script, which GET's /connect-client which returns the initial response, and the infinite while loop runs the send_message - forever.

However - subsequent GET's to /client (by refreshing the browser), fail - is this caused by the maximum number of connected clients issue, or is this a browser issue https://developer.mozilla.org/en-US/docs/Web/API/EventSource ?

Here's the traceback when refreshing the browser:

0;🐍192.168.178.89 | code.py | 8.2.0-rc.1Started development server on http://192.168.178.89:80
192.168.178.74 -- "GET /connect-client" 408 -- "200 OK" 101
192.168.178.74 -- "GET /client" 510 -- "200 OK" 474
Traceback (most recent call last):
  File "/lib/adafruit_httpserver/server.py", line 351, in poll
  File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request
  File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler
  File "code.py", line 60, in connect_client
  File "/lib/adafruit_httpserver/response.py", line 483, in close
  File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes
BrokenPipeError: 32
Traceback (most recent call last):
  File "code.py", line 67, in <module>
  File "/lib/adafruit_httpserver/server.py", line 380, in poll
  File "/lib/adafruit_httpserver/server.py", line 351, in poll
  File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request
  File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler
  File "code.py", line 60, in connect_client
  File "/lib/adafruit_httpserver/response.py", line 483, in close
  File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes
BrokenPipeError: 32
0;🐍192.168.178.89 | 117@/lib/adafruit_httpserver/respons BrokenPipeError | 8.2.0-rc.1
Code done running.

def connect_client(request: Request):
    response = SSEResponse(request)

    if connected_client.response is not None:
        connected_client.response.close()  # Close any existing connection  <<<= line 60
    connected_client.response = response
    return response

The connected_client.response is obviously not None but it's in some state that does not allow it to be closed. CircuitPython is happy to recover by soft rebooting - but ...

[michalpokusa]

I am unable to replicate thsi error, which browser are you using? It also might be the Pico thing, I am using ESP32-S2 Feather and it simply switches to new client.

EDIT1: Also, I am using stable CP 8.1.0, try with this version, maybe something changed in 8.2.0.

[adrianblakey]

It's lovely Chrome Version 114.0.5735.198 (Official Build) (64-bit) on ArchLinux _ suppose I could give it a shot w 8.1.0 and report back ...

[adrianblakey]

With 8.1.0 - it now behaves differently

The REPL ...

0;🐍192.168.178.89 | code.py | 8.1.0Started development server on http://192.168.178.89:80                                                                        
192.168.178.74 -- "GET /connect-client" 408 -- "200 OK" 101                                                                                                       
connected client response <SSEResponse object at 0x2002eda0>                                                                                                      
192.168.178.74 -- "GET /connect-client" 408 -- "200 OK" 101                                                                                                       
connected client response <SSEResponse object at 0x2001aef0>  
...

and in the browser console ...

[michalpokusa]

So to make this clear, you try to connect multiple clients, and there is no error on server side, but on both clients you get errors?

The CONNECTION_REFUSED is a bit diffrent from what i expected, but since example allows only one client and switches between them, it is normal that clients throw erros, as they in fact are constantly disconnected and connected again, like they are fighting for their place.

The output I get is on both clients is 2-3 "Event data:...", then error, and then it repeats.

Nevertheless, it kind of proves that there was in fact a change in CP itself.

[adrianblakey]

I reran the test again. I noticed I had some tabs open before that were also submitting the requests thanks to Chrome's persistence...

The test is:

• ctl-D to start the code.py
• http://192.168.178.89/client (F12 to open the console display in the browser) - events appear correctly
• Open new browser tab
• Run another http://192.168.178.89/client
• Browser console errors: SSE error: Event {isTrusted: true, type: 'error', target: EventSource, currentTarget: EventSource, eventPhase: 2, …} every 2/3 responses in both browser tabs
• REPL shows ... 192.168.178.74 -- "GET /connect-client" 378 -- "200 OK" 101
  connected client response <SSEResponse object at 0x20026330>
  192.168.178.74 -- "GET /connect-client" 408 -- "200 OK" 101
  connected client response <SSEResponse object at 0x2002cc60>

If I repeat the same test of refreshing the browser it behaves the same way as 8.2.0 :-)

192.168.178.74 -- "GET /client" 506 -- "200 OK" 474                                                                                                               
192.168.178.74 -- "GET /connect-client" 430 -- "200 OK" 101                                                                                                       
192.168.178.74 -- "GET /client" 510 -- "200 OK" 474                                                                                                               
connected client response <SSEResponse object at 0x2001da00>                                                                                                      
Traceback (most recent call last):                                                                                                                                
  File "/lib/adafruit_httpserver/server.py", line 351, in poll                                                                                                    
  File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request                                                                                         
  File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler                                                                                          
  File "code.py", line 61, in connect_client                                                                                                                      
  File "/lib/adafruit_httpserver/response.py", line 483, in close                                                                                                 
  File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes                                                                                           
BrokenPipeError: 32                                                                                                                                               
Traceback (most recent call last):                                                                                                                                
  File "code.py", line 68, in <module>                                                                                                                            
  File "/lib/adafruit_httpserver/server.py", line 380, in poll                                                                                                    
  File "/lib/adafruit_httpserver/server.py", line 351, in poll                                                                                                    
  File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request                                                                                         
  File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler                                                                                          
  File "code.py", line 61, in connect_client                                                                                                                      
  File "/lib/adafruit_httpserver/response.py", line 483, in close                                                                                                 
  File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes                                                                                           
BrokenPipeError: 32                                                                                                                                               
0;🐍192.168.178.89 | 117@/lib/adafruit_httpserver/respons BrokenPipeError | 8.1.0                                                                                 
Code done running.                                                                                                                                                

Press any key to enter the REPL. Use CTRL-D to reload.  

[michalpokusa]

If every browser tabs error every 2/3 successfull event data logs, then it works as expected, because as I said before, the clients are both fighting for the only one space.

When it comes to BrokenPipeError, I cannot replicate it. It may be browser, CP or Pico related, do you have any other microcontroller you can test that on?

[adrianblakey]

Is there something I could do to gather more data about the BrokenPipeError and debug it? This is the only mcu I have.

[michalpokusa]

Nothing comes to my mind right now.

Considering we are both using the same code, browser and CP version, I say it is a CP implementation thing. It would not be the first time, in the past there were also some diffrences in socket behavior between ESPs and Pico.

In my opinion, you could make an issue on CP itself about that, it probably is causesd by missing the close on socket or something similar.

Another way, although a bit hacky, is supressing this error, of course only it your program works doing so.

[adrianblakey]

btw - here's the Firefox behavior

0;🐍192.168.178.89 | code.py | 8.1.0Started development server on http://192.168.178.89:80                                                                          
192.168.178.74 -- "GET /client" 344 -- "200 OK" 474                                                                                                                 
192.168.178.74 -- "GET /connect-client" 336 -- "200 OK" 101                                                                                                         
192.168.178.74 -- "GET /favicon.ico" 298 -- "404 Not Found" 90                                                                                                      
192.168.178.74 -- "GET /client" 344 -- "200 OK" 474                                                                                                                 
connected client response <SSEResponse object at 0x2001bb40>                                                                                                        
Traceback (most recent call last):                                                                                                                                  
  File "/lib/adafruit_httpserver/server.py", line 351, in poll                                                                                                      
  File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request                                                                                           
  File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler                                                                                            
  File "code.py", line 61, in connect_client                                                                                                                        
  File "/lib/adafruit_httpserver/response.py", line 483, in close                                                                                                   
  File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes                                                                                             
OSError: [Errno 9] EBADF                                                                                                                                            
Traceback (most recent call last):                                                                                                                                  
  File "code.py", line 68, in <module>                                                                                                                              
  File "/lib/adafruit_httpserver/server.py", line 380, in poll                                                                                                      
  File "/lib/adafruit_httpserver/server.py", line 351, in poll                                                                                                      
  File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request                                                                                           
  File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler                                                                                            
  File "code.py", line 61, in connect_client                                                                                                                        
  File "/lib/adafruit_httpserver/response.py", line 483, in close                                                                                                   
  File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes                                                                                             
OSError: [Errno 9] EBADF                                                                                                                                            
0;🐍192.168.178.89 | 117@/lib/adafruit_httpserver/respons OSError | 8.1.0                                                                                           
Code done running.                                                                                                                                                  

Press any key to enter the REPL. Use CTRL-D to reload.        

[michalpokusa]

Try wrapping the _close_connection in SSEResponse or the body od _close_connection itself in try except:

def _close_connection(self) -> None:
    try:
        self._request.connection.close()
    except:
        pass

Maybe Pico closes the socket on it's own or other dark magic causes this. ¯\(ツ)/¯

[adrianblakey]

Back on 8.2.0 rc1. Yes the connection does get closed ... so this seems to do the trick

    def _close_connection(self) -> None:
        try:
            self._request.connection.close()
        except (BrokenPipeError, OSError) as e:  # Client closed the connection already
            pass

and

    def close(self):
        """
        Close the connection.

        **Always call this method when you are done sending events.**
        """

        try:
            self._send_bytes(self._request.connection, b"event: close\n")
            self._close_connection()
        except (BrokenPipeError, OSError) as e:     # Client closed the connection already
            pass

and on send_event - about line 477

        try:
            self._send_bytes(self._request.connection, message.encode("utf-8"))
        except (BrokenPipeError, OSError) as e:          # Might fail due to another open browser tab
            pass

Interesting behavior -

• Start 1 tab.
• Refresh this tab - all is now fine.
• Start another tab - crashes the VM, which does a soft reboot and services the new tab - not the old.
• Refresh the first and second tab and they now (sort of) cooperate emitting say 2-3 events each before giving an error

How do we get the VM crash fixed?

[michalpokusa]

What do you mean by VM?

[adrianblakey]

The python interpreter.

When the code has an issue or the interpreter has an issue doesn't it reboot (which is what happens the first time new tab is opened and the request submitted)? Maybe there is another issue with python code that throws a runtime excp that's not being caught and I am mistaken ...

Also, is there any way to speed up the rate at which the events are fired (send_message is called)? Seems like it occurs about once a second - I need this at about every 1/10 sec. Could I go back to playing games with asyncio, caching say 10 events and emitting them from the cache in send_message?

[michalpokusa]

Regarding more frequent messages, yes, you can increase that, line 28 on example:

self.next_message = monotonic() + <time in seconds here>