Curl-cffi is a great library that mimics curl requests but with SSL fingerprint spoofing to make it look like the request is coming from a real chrome or safari or firefox browser.
Up until recently I was fine running curl-cffi (which has an async API) with trio using trio-asyncio. However recently I am getting a bunch of BusyResourceError errors:
Exception in default exception handler
Traceback (most recent call last):
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1797, in call_exception_handler
self.default_exception_handler(context)
File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio_asyncio/_async.py", line 44, in default_exception_handler
raise exception
File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio_asyncio/_base.py", line 502, in _reader_loop
await _wait_readable(fd)
File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio/_core/_generated_io_epoll.py", line 45, in wait_readable
return await GLOBAL_RUN_CONTEXT.runner.io_manager.wait_readable(fd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio/_core/_io_epoll.py", line 336, in wait_readable
await self._epoll_wait(fd, "read_task")
File "/home/laurens/projects/news-archive/venv/lib/python3.11/site-packages/trio/_core/_io_epoll.py", line 300, in _epoll_wait
raise _core.BusyResourceError(
trio.BusyResourceError: another task is already reading / writing this fd
I am doing nothing complicated, just the basic AsyncSession and request pattern in parallel, with some cancel scopes wrapped around. So not much more complicated than this:
from curl_cffi.requests import AsyncSession
from trio_asyncio import aio_as_trio
import trio
# Many of the below in parallel
async with AsyncSession() as s:
with trio.move_on_after(1):
r = await aio_as_trio(s.get)("https://example.com")
When I debug the error by putting a breakpoint in default_exception_handler, it always has to do with a cancelled cancelscope handle:
pprint(context)
{'exception': BusyResourceError('another task is already reading / writing this fd'),
'handle': <ScopedHandle cancelled scope=<trio.CancelScope at 0x14ae9ed2cd60, active, cancelled>>,
'message': 'Exception in callback None()'}
Curl-cffi is a great library that mimics curl requests but with SSL fingerprint spoofing to make it look like the request is coming from a real chrome or safari or firefox browser.
Up until recently I was fine running curl-cffi (which has an async API) with trio using trio-asyncio. However recently I am getting a bunch of BusyResourceError errors:
I am doing nothing complicated, just the basic AsyncSession and request pattern in parallel, with some cancel scopes wrapped around. So not much more complicated than this:
When I debug the error by putting a breakpoint in
default_exception_handler
, it always has to do with a cancelled cancelscope handle: