aio-libs / aiohttp

Asynchronous HTTP client/server framework for asyncio and Python
https://docs.aiohttp.org
Other
14.97k stars 2k forks source link

Regression in aiohttp 3.10.3 when running multiple loops in multiple threads #9202

Closed hauntsaninja closed 2 hours ago

hauntsaninja commented 3 hours ago

Describe the bug

λ uv pip show aiohttp
Name: aiohttp
Version: 3.10.2
Location: .../lib/python3.11/site-packages
Requires: aiohappyeyeballs, aiosignal, attrs, frozenlist, multidict, yarl
Required-by: adlfs, aiogoogle, aiohttp-cors, aiohttp-retry, aiomonitor, aioresponses, aiosseclient, apache-airflow-providers-http, boostedblob, datasets, geoip2, kubernetes-asyncio, langchain, langchain-community, livekit-api, microsoft-kiota-authentication-azure, pytest-aiohttp, safety, twilio

λ cat a.py
import asyncio
import aiohttp
import concurrent.futures

async def aiohttp_get():
    async with aiohttp.ClientSession() as session:
        async with session.get("https://httpbin.org/get") as response:
            return await response.text()

def sync():
    asyncio.run(aiohttp_get())

futs = []
with concurrent.futures.ThreadPoolExecutor() as executor:
    for _ in range(10):
        futs.append(executor.submit(sync))
for fut in futs:
    fut.result()

# works fine with aiohttp==3.10.2
λ python a.py

λ uv pip install 'aiohttp==3.10.3'
Resolved 8 packages in 51ms
Uninstalled 1 package in 11ms
Installed 1 package in 5ms
 - aiohttp==3.10.2
 + aiohttp==3.10.3

# crashes with aiohttp==3.10.3 and newer
λ python a.py
Traceback (most recent call last):
  File "/Users/shantanu/devoai/openai/a.py", line 21, in <module>
    fut.result()
  File "/Users/shantanu/.pyenv/versions/3.11.8/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/shantanu/.pyenv/versions/3.11.8/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/shantanu/.pyenv/versions/3.11.8/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shantanu/devoai/openai/a.py", line 13, in sync
    asyncio.run(aiohttp_get())
  File "/Users/shantanu/.pyenv/versions/3.11.8/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/shantanu/.pyenv/versions/3.11.8/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shantanu/.pyenv/versions/3.11.8/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/shantanu/devoai/openai/a.py", line 8, in aiohttp_get
    async with session.get("https://httpbin.org/get") as response:
  File ".../lib/python3.11/site-packages/aiohttp/client.py", line 1353, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/aiohttp/client.py", line 657, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/aiohttp/connector.py", line 564, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/aiohttp/connector.py", line 975, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/aiohttp/connector.py", line 1285, in _create_direct_connection
    sslcontext = await self._get_ssl_context(req)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/aiohttp/connector.py", line 1028, in _get_ssl_context
    return await self._make_or_get_ssl_context(True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/aiohttp/connector.py", line 1033, in _make_or_get_ssl_context
    return await self._made_ssl_context[verified]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Task <Task pending name='Task-2' coro=<aiohttp_get() running at /Users/shantanu/devoai/openai/a.py:8> cb=[_run_until_complete_cb() at /Users/shantanu/.pyenv/versions/3.11.8/lib/python3.11/asyncio/base_events.py:181]> got Future <Future pending> attached to a different loop

To Reproduce

See a.py in log above

Expected behavior

No crash

Logs/tracebacks

See above

Python Version

3.11.8

aiohttp Version

3.10.3 and newer

multidict Version

6.1

yarl Version

1.11.1

OS

macOS

Related component

Client

Code of Conduct

Dreamsorcerer commented 3 hours ago

9020?

hauntsaninja commented 2 hours ago

Yup, can confirm that that fixes! Thank you (and thank you bdraco for the fix). I'll wait for 3.10.6 then :-)

(I missed this when searching for issues because I searched phrases from stacktrace, not future attached to a different loop because that has a lot of other hits)