Client ECONNRESET errors when POSTing to micropython microdot from micropython aiohttp

[ned-pcs]

See https://github.com/miguelgrinberg/microdot/issues/201

[miguelgrinberg]

I think this is an issue in MicroPython's asyncio library.

In the example code posted by the OP microdot returns the string "OK" as a response. The client calls response.text() to get the body of the response from the response object returned by aiohttp, and text() calls read(-1) on the stream.

Take a look at the read() implementation for asyncio streams:

    # async
    def read(self, n=-1):
        r = b""
        while True:
            yield core._io_queue.queue_read(self.s)
            r2 = self.s.read(n)
            if r2 is not None:
                if n >= 0:
                    return r2
                if not len(r2):
                    return r
                r += r2

Now maybe I'm missing something here, but considering that the server writes "OK" and then closes, the above code will read "OK" into r2 in the first iteration of the while-loop and skip the return because n == -1, then set r (which is empty at this point) to r2 before going back up for a second iteration.

In the second iteration the read call raises ECONNRESET because the server moved on to other things, and this error is not caught. I think this error (probably also EPIPE) should be silenced and r should be returned if at least one byte has been read already. Don't you think?

Also aside from this, I think the aiohttp library should not issue a read(-1) when the response returned a Content-Length header. In that case it should parse this header and issue a read for the exact body length. This would also prevent the issue reported by the OP, I think.

[Carglglz]

Also aside from this, I think the aiohttp library should not issue a read(-1) when the response returned a Content-Length header. In that case it should parse this header and issue a read for the exact body length. This would also prevent the issue reported by the OP, I think.

That was it, I've just updated PR #782 with the fix, so it should work now 👍🏼

[ned-pcs]

After updating the ESP32 client's aiohttp with the new code, I'm still seeing the ECONNRESET error with Unix micropython running the microdot server, but not with desktop Python3.

[Carglglz]

@ned-pcs , I've tested both scripts with unix port and they work now, so I will try to test with ESP32 port too, but I see no reason why it shouldn't work 🤔 ....

[ned-pcs]

@Carglglz The Unix port (client and server) works for me, the ESP32 client with Unix micropython server still reports ECONNRESET (now in readline() instead of in read()). I've even tried editing extmod/asyncio/stream.py as @miguelgrinberg suggested:

from errno import ECONNRESET

try:
    from errno import EPIPE
except ImportError:
    EPIPE = 32

...

    # async
    def read(self, n=-1):
        r = b""
        while True:
            try:
                yield core._io_queue.queue_read(self.s)
                r2 = self.s.read(n)
            except OSError as exc:
                if exc.errno in (ECONNRESET, EPIPE) and len(r):
                    return r
                raise
            if r2 is not None:
                if n >= 0:
                    return r2
                if not len(r2):
                    return r
                r += r2

...

    # async
    def readline(self):
        l = b""
        while True:
            try:
                yield core._io_queue.queue_read(self.s)
                l2 = self.s.readline()  # may do multiple reads but won't block
            except OSError as exc:
                if exc.errno in (ECONNRESET, EPIPE) and len(l):
                    return l
                raise
            if l2 is None:
                continue
            l += l2
            if not l2 or l[-1] == 10:  # \n (check l in case l2 is str)
                return l

[ned-pcs]

OK, if I change the logic in extmod/asyncio/stream.py to return possibly-empty strings upon ECONNRESET things work on the ESP32:

    # async
    def read(self, n=-1):
        r = b""
        while True:
            try:
                yield core._io_queue.queue_read(self.s)
                r2 = self.s.read(n)
            except OSError as exc:
                if exc.errno in (ECONNRESET, EPIPE):
                    return r
                raise
            if r2 is not None:
                if n >= 0:
                    return r2
                if not len(r2):
                    return r
                r += r2

...

    # async
    def readline(self):
        l = b""
        while True:
            try:
                yield core._io_queue.queue_read(self.s)
                l2 = self.s.readline()  # may do multiple reads but won't block
            except OSError as exc:
                if exc.errno in (ECONNRESET, EPIPE):
                    return l
                raise
            if l2 is None:
                continue
            l += l2
            if not l2 or l[-1] == 10:  # \n (check l in case l2 is str)
                return l

[Carglglz]

@ned-pcs please test again #782 with the latest changes, I've just tested it and it works now so I think this https://github.com/micropython/micropython/pull/13478 can be closed 👍🏼

[Carglglz]

After updating the ESP32 client's aiohttp with the new code, I'm still seeing the ECONNRESET error with Unix micropython running the microdot server, but not with desktop Python3.

@ned-pcs , after some further analysis my conclusions are that Stream.drain() behaves differently in MicroPython. The reason is that the query that included data i.e. POST method, was wrongly implemented adding extra b'\r\n' bytes at the end.

This data was left unread by the server, which seems to cause the socket to be closed promptly after calling Stream.drain().

The difference between unix and esp32 ports is that unix is faster so it could read everything before the socket was closed (I tested this adding await asyncio.sleep(1) before calling await writer.aclose() which made it work for esp32 port too).

[ned-pcs]

@Carglglz thanks for your work on this.