search

encode / httpx

A next generation HTTP client for Python. 🦋
https://www.python-httpx.org/
BSD 3-Clause "New" or "Revised" License
13.28k stars 844 forks source link

Threading race condition in _eventloop.get_asynclib() (with fix) #2116

Closed danielrobbins closed 2 years ago

danielrobbins commented 2 years ago

I have a threading Web spider that is triggering a race condition in anyio/_core/_eventloop.py's get_asynclib() function. This race can be triggered if you have a lot of threads trying to initialize their own asyncio event loop and initializing httpx all at once. It happens intermittently -- sometimes we don't trigger the race, sometimes we do. When we do, the threads catch get_asynclib() mid-initialization and certain attributes that should be present are not available. I am using httpx 0.22.0 and python3.7.

httpx-exception.txt

An ugly but functioning patch which I can confirm resolves the issue is below:

get_asynclib_thread_fix.patch.txt

In my test patch, I made the lock cover essentially the entire get_asynclib() call -- it could potentially be made tighter -- not sure.

Thanks for creating httpx -- it's very nice! :)

danielrobbins commented 2 years ago

Sorry, I was rushing :) This was a bug in anyio. I will file with them.

danielrobbins commented 2 years ago

Opened the same bug with anyio here: https://github.com/agronholm/anyio/issues/425