search

aio-libs / aiohttp

Asynchronous HTTP client/server framework for asyncio and Python
https://docs.aiohttp.org
Other
15.11k stars 2.02k forks source link

RuntimeError: no running event loop #8555

Closed tomplus closed 3 months ago

tomplus commented 3 months ago

Describe the bug

It looks like a breaking change in the latest version.

In version 3.9.5 it works:

import aiohttp
connector = aiohttp.TCPConnector()

(in a more complex example I get a warning (DeprecationWarning: The object should be created within an async function))

but in 3.10 I get an exception:

Traceback (most recent call last):
  File "/tmp/a1.py", line 5, in <module>
    connector = aiohttp.TCPConnector()
  File "/home/tomek/work/github/kubernetes-client/kubernetes_asyncio-new/VENV/lib/python3.10/site-packages/aiohttp/connector.py", line 784, in __init__
    super().__init__(
  File "/home/tomek/work/github/kubernetes-client/kubernetes_asyncio-new/VENV/lib/python3.10/site-packages/aiohttp/connector.py", line 234, in __init__
    loop = loop or asyncio.get_running_loop()
RuntimeError: no running event loop

To Reproduce

code

import aiohttp
connector = aiohttp.TCPConnector()

Expected behavior

If the change is intentional it would be very useful to describe that in a changelog. If not I expect the same behavior like in the previous version.

Logs/tracebacks

Traceback (most recent call last):
  File "/tmp/a1.py", line 5, in <module>
    connector = aiohttp.TCPConnector()
  File "/home/tomek/work/github/kubernetes-client/kubernetes_asyncio-new/VENV/lib/python3.10/site-packages/aiohttp/connector.py", line 784, in __init__
    super().__init__(
  File "/home/tomek/work/github/kubernetes-client/kubernetes_asyncio-new/VENV/lib/python3.10/site-packages/aiohttp/connector.py", line 234, in __init__
    loop = loop or asyncio.get_running_loop()
RuntimeError: no running event loop

### Python Version

```console
$ python --version
3.10

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.10.0

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.4

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.8.2

OS

Linux

Related component

Client

Additional context

No response

Code of Conduct

bdraco commented 3 months ago

It looks like this is caused by #8512 which is a backport of https://github.com/aio-libs/aiohttp/pull/5278 from master

bdraco commented 3 months ago

It looks like it was deprecated in 3.9.5 and now no longer works in 3.10.0 https://github.com/aio-libs/aiohttp/blob/3.9/aiohttp/helpers.py#L296 but https://github.com/aio-libs/aiohttp/pull/5278 didn't have a changelog message to say it would stop working.

In 3.9.5 it would create a new non-running event loop

>>> import aiohttp
>>> print(aiohttp.__version__)
3.9.5
>>> connector = aiohttp.TCPConnector()
>>> print(connector._loop)
<_UnixSelectorEventLoop running=False closed=False debug=False>
>>>
bdraco commented 3 months ago

Based on the above, I'd say the change is intentional but could use a changelog entry.

bdraco commented 3 months ago

Creating a ClientSession, CookieJar, or Connector without a running asyncio event loop will now explicitly raise in 3.10.0 because it calls asyncio.get_running_loop()

aslonnie commented 3 months ago

is this going to be fixed in 3.10.1 ? or is this breakage intentional?

bdraco commented 3 months ago

The only planned change is to improve the changelog. As it was deprecated over 5 years ago, there is no plan to extend the deprecation period.

Edit: This practice is deprecated in Python as well

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_event_loop

Screenshot 2024-08-01 at 2 27 12 PM
TheDurableDane commented 3 months ago

This is a breaking change (intentional or not), so shouldn't the version be bumped to 4.0.0? Regardless of the length of the deprecation period.

webknjaz commented 3 months ago

https://hynek.me/articles/semver-will-not-save-you/#taking-responsibility

TheDurableDane commented 3 months ago

@webknjaz That's a bit passive-aggressive. I know I shouldn't blindly trust any package I use, but sematic versioning best pratices would be a nice touch for such a widely used package as aiohttp. I don't think you are excused just because you can link to a blog post that says it's the user's fault.

bdraco commented 3 months ago

I thought about a partial revert of #8512 (originally #5278) on 3.10/3.11, but I'm not sold on that idea because Python has deprecated calling asyncio.get_event_loop() without a running event loop. After all, the behavior is confusing. Even if we did revert this, it's kicking the can down the road until it becomes an error.

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_event_loop

Screenshot 2024-08-01 at 2 27 12 PM
webknjaz commented 3 months ago

@TheDurableDane you're right the end-users should be able to rely on the documented expectations. Could you point out where we promise that the first version component alone corresponds to SemVer's major version? My observation is that major has always two components. Besides, SemVer is talking about public APIs and my understanding is that those haven't changed. And finally, people who make releases may interpret things differently, compared to those consuming them. I keep seeing this problem where the users invent something in their heads and think this always matches the mental models of others. That's not how it works. SemVer by its nature ultimately represents how the maintainers see changes, not necessarily how every single user perceive them.

Dreamsorcerer commented 3 months ago

Just to add to bdraco's conclusion, I think the old code may have been dangerously wrong in some situations. If you create TCPConnector as a global, the get_event_loop() call would create a new loop and store the reference for later. If you then run your application with asyncio.run() or web.run_app(), it will create a new loop to execute the application in. You then have your application running in one loop and TCPConnector trying to do callbacks etc. in another loop which isn't even running.

bdraco commented 3 months ago

0001-Restore-the-ability-to-create-objects-without-a-runn.patch

Attached is a patch for anyone who arrives at this issue in the future because a security issue requires them to update to 3.10.x or later and needs a quick solution until they can adjust their object creation.

kyrlon commented 1 month ago

0001-Restore-the-ability-to-create-objects-without-a-runn.patch

Attached is a patch for anyone who arrives at this issue in the future because a security issue requires them to update to 3.10.x or later and needs a quick solution until they can adjust their object creation.

Can confirm this works on my Windows 10 machine:

OS: Windows 10
Python version: Python3.12
aiohttp==3.10.5

however got this working by manual alterations.

  1. I cannot for the life of me build from source (or pip install via py -m pip install git+https://github.com/aio-libs/aiohttp.git), so that was my first issue

    Note: Here is the log on the build attempt... 
    
Collecting git+https://github.com/aio-libs/aiohttp.git
Cloning https://github.com/aio-libs/aiohttp.git to c:\users\kyrlon\appdata\local\temp\1\pip-req-build-n53k6a8a
Running command git clone --filter=blob:none --quiet https://github.com/aio-libs/aiohttp.git 'C:\Users\kyrlon\AppData\Local\Temp\1\pip-req-build-n53k6a8a'
Resolved https://github.com/aio-libs/aiohttp.git to commit ffcf9dc4ea157adc5b7b5b31b6cc69f37d533122
Running command git submodule update --init --recursive -q
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting aiohappyeyeballs>=2.3.0 (from aiohttp==4.0.0a2.dev0)
Using cached aiohappyeyeballs-2.4.0-py3-none-any.whl (12 kB)
Collecting aiosignal>=1.1.2 (from aiohttp==4.0.0a2.dev0)
Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting frozenlist>=1.1.1 (from aiohttp==4.0.0a2.dev0)
Using cached frozenlist-1.4.1-cp311-cp311-win_amd64.whl (50 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp==4.0.0a2.dev0)
Using cached multidict-6.1.0-cp311-cp311-win_amd64.whl (28 kB)
Collecting yarl<2.0,>=1.11.0 (from aiohttp==4.0.0a2.dev0)
Using cached yarl-1.11.1-cp311-cp311-win_amd64.whl (110 kB)
Requirement already satisfied: idna>=2.0 in c:\users\kyrlon\appdata\local\programs\python\python311\lib\site-packages (from yarl<2.0,>=1.11.0->aiohttp==4.0.0a2.dev0) (3.7)
Building wheels for collected packages: aiohttp
Building wheel for aiohttp (pyproject.toml): started
Building wheel for aiohttp (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error

Building wheel for aiohttp (pyproject.toml) did not run successfully.
exit code: 1

[90 lines of output]
*********************
* Accelerated build *
*********************
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-311
creating build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\abc.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\base_protocol.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client_exceptions.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client_proto.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client_reqrep.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client_ws.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\compression_utils.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\connector.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\cookiejar.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\formdata.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\hdrs.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\helpers.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http_exceptions.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http_parser.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http_websocket.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http_writer.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\locks.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\log.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\multipart.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\payload.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\pytest_plugin.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\resolver.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\streams.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\tcp_helpers.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\test_utils.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\tracing.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\typedefs.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_app.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_exceptions.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_fileresponse.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_log.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_middlewares.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_protocol.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_request.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_response.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_routedef.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_runner.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_server.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_urldispatcher.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_ws.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\worker.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\__init__.py -> build\lib.win-amd64-cpython-311\aiohttp
running egg_info
writing aiohttp.egg-info\PKG-INFO
writing dependency_links to aiohttp.egg-info\dependency_links.txt
writing requirements to aiohttp.egg-info\requires.txt
writing top-level names to aiohttp.egg-info\top_level.txt
reading manifest file 'aiohttp.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'aiohttp' anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.lib' found anywhere in distribution
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.a' found anywhere in distribution
warning: no previously-included files matching '*.obj' found anywhere in distribution
warning: no previously-included files found matching 'aiohttp\*.html'
no previously-included directories found matching 'docs\_build'
adding license file 'LICENSE.txt'
writing manifest file 'aiohttp.egg-info\SOURCES.txt'
copying aiohttp\_cparser.pxd -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\_find_header.pxd -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\_helpers.pyi -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\_helpers.pyx -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\_http_parser.pyx -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\_http_writer.pyx -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\_websocket.pyx -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\py.typed -> build\lib.win-amd64-cpython-311\aiohttp
running build_ext
building 'aiohttp._websocket' extension
creating build\temp.win-amd64-cpython-311
creating build\temp.win-amd64-cpython-311\Release
creating build\temp.win-amd64-cpython-311\Release\aiohttp
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\kyrlon\AppData\Local\Programs\Python\Python311\include -IC:\Users\kyrlon\AppData\Local\Programs\Python\Python311\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" /Tcaiohttp/_websocket.c /Fobuild\temp.win-amd64-cpython-311\Release\aiohttp/_websocket.obj
_websocket.c
c1: fatal error C1083: Cannot open source file: 'aiohttp/_websocket.c': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for aiohttp
Failed to build aiohttp
ERROR: Could not build wheels for aiohttp, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 23.1.2 -> 24.2 [notice] To update, run: python.exe -m pip install --upgrade pip

  1. At the time of this post, I was using aiohttp==3.10.5, so when I did attempt to patch, it failed. Manual modifications was a slow and patient process (inspecting the patch with a text editor), but afterwards was able run such an example on my current system mentioned from this article: 
    
import sys
import os
import json
import asyncio
import aiohttp

Initialize connection pool

conn = aiohttp.TCPConnector(limit_per_host=100, limit=0, ttl_dns_cache=300) PARALLEL_REQUESTS = 100 results = [] urls = ['https://jsonplaceholder.typicode.com/todos/1' for i in range(4000)] #array of urls

async def gather_with_concurrency(n): semaphore = asyncio.Semaphore(n) session = aiohttp.ClientSession(connector=conn)

# heres the logic for the generator
async def get(url):
    async with semaphore:
        async with session.get(url, ssl=False) as response:
            obj = json.loads(await response.read())
            results.append(obj)
await asyncio.gather(*(get(url) for url in urls))
await session.close()

if name == "main": loop = asyncio.get_event_loop() loop.run_until_complete(gather_with_concurrency(PARALLEL_REQUESTS)) conn.close()

print(f"Completed {len(urls)} requests with {len(results)} results")
zeocax commented 1 month ago

This may help

import aiohttp
import asyncio

async def create_session():
    return aiohttp.ClientSession()

aiohttp_client_session = asyncio.get_event_loop().run_until_complete(create_session())
Dreamsorcerer commented 1 month ago

As mentioned above, very likely to break your application (now or in the future).