python-trio / trio-asyncio

a re-implementation of the asyncio mainloop on top of Trio
Other
189 stars 38 forks source link

Incompatible with curl-cffi library: trio.BusyResourceError: another task is already reading / writing this fd #153

Closed iiLaurens closed 2 months ago

iiLaurens commented 2 months ago

Curl-cffi is a great library that mimics curl requests but with SSL fingerprint spoofing to make it look like the request is coming from a real chrome or safari or firefox browser.

Up until recently I was fine running curl-cffi (which has an async API) with trio using trio-asyncio. However recently I am getting a bunch of BusyResourceError errors:

Exception in default exception handler
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1797, in call_exception_handler
    self.default_exception_handler(context)
  File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio_asyncio/_async.py", line 44, in default_exception_handler
    raise exception
  File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio_asyncio/_base.py", line 502, in _reader_loop
    await _wait_readable(fd)
  File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio/_core/_generated_io_epoll.py", line 45, in wait_readable
    return await GLOBAL_RUN_CONTEXT.runner.io_manager.wait_readable(fd)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio/_core/_io_epoll.py", line 336, in wait_readable
    await self._epoll_wait(fd, "read_task")
  File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio/_core/_io_epoll.py", line 300, in _epoll_wait
    raise _core.BusyResourceError(
trio.BusyResourceError: another task is already reading / writing this fd

I am doing nothing complicated, just the basic AsyncSession and request pattern in parallel, with some cancel scopes wrapped around. So not much more complicated than this:

from curl_cffi.requests import AsyncSession
from trio_asyncio import aio_as_trio
import trio

# Many of the below in parallel
async with AsyncSession() as s:
    with trio.move_on_after(1):
        r = await aio_as_trio(s.get)("https://example.com")

When I debug the error by putting a breakpoint in default_exception_handler, it always has to do with a cancelled cancelscope handle:

pprint(context)
{'exception': BusyResourceError('another task is already reading / writing this fd'),
 'handle': <ScopedHandle cancelled scope=<trio.CancelScope at 0x14ae9ed2cd60, active, cancelled>>,
 'message': 'Exception in callback None()'}