Windows - Python 3.8+ raises "RuntimeError: Event Loop is closed" on exit.

rmawatson commented 4 years ago

Checklist

[ x] The bug is reproducible against the latest release and/or master.
[ x] There are no similar issues or pull requests to fix it yet.

Describe the bug

I get the following traceback when running the attached code

404
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000017C63D5E310>
Traceback (most recent call last):
  File "C:\Python\Python38\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Python\Python38\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Python\Python38\lib\asyncio\base_events.py", line 715, in call_soon
    self._check_closed()
  File "C:\Python\Python38\lib\asyncio\base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

To reproduce

async def testing():
    client = httpx.AsyncClient()
    response = await client.get("https://www.google.com/")
    await client.aclose()
    print(response.status_code)

async def main():
   await testing()

asyncio.run(main())

Inserting time.sleep(0.1) at the end of main() fixes it. Possibly due to the same reasons in the issues linked below.

Environment

Windows 10 64bit
Python version: 3.8.1
HTTPX version 0.12.1

Possibly the same as this and this

florimondmanca commented 4 years ago

I assume this is yet another strange issue with asyncio’s stream.close() - it schedules the closing of the socket but it’s not guaranteed to be done once client.aclose finishes, which means the socket is actually deleted upon garbage collection, and by then I suppose the event loop is already closed... The sleep() call fixed this because it allows asyncio to execute the close callback before proceeding to shutdown the async environment.

See also #825 for more discussion on a possibly related issue.

Have you tried running this on Python 3.8.2?

tomchristie commented 4 years ago

Also: May well be specific to Window's Proactor event loop.

florimondmanca commented 4 years ago

Yes, definitely looks like a Windows specific issue to me - the linked issues are for Windows too.

rmawatson commented 4 years ago

@florimondmanca , yes upgraded to 3.8.2 to check, the result is the same.

Baysul commented 4 years ago

Can confirm this is an issue with the latest PyPi version of aiohttp and Python 3.8.2 on Windows.

florimondmanca commented 4 years ago

Okay, thank you.

I personally don’t have a Windows machine so help with investigation and possible fixes would help.

Would we need to consider black magic such as https://github.com/aio-libs/aiohttp/issues/1925#issuecomment-592596034? How about https://github.com/aio-libs/aiohttp/pull/3733?

FYI this is likely to be something to address on the HTTPCore side, as connection management and networking is done there. Might then be helpful to first come up with a minimal repro example using HTTPCore only?

tomchristie commented 4 years ago

It'd also be helpful to try switching to using SelectorEventLoop and let us know what the behaviour is like then... https://docs.python.org/3/library/asyncio-eventloop.html#event-loop-implementations (Or to try with Python 3.7 or earlier)

Python only switched the default event loop to proactor from 3.8 onwards. https://docs.python.org/3/library/asyncio-policy.html#asyncio.DefaultEventLoopPolicy

euri10 commented 4 years ago

not sure there's a link and can't remember why now but in uvicorn we force the SelectorEventLoop to be the default for windows

Le ven. 1 mai 2020 à 10:23 AM, Tom Christie notifications@github.com a écrit :

It'd also be helpful to try switching to using SelectorEventLoop and let us know what the behaviour is like then... https://docs.python.org/3/library/asyncio-eventloop.html#event-loop-implementations (Or to try with Python 3.7 or earlier)

Python only switched the default event loop to proactor from 3.8 onwards. https://docs.python.org/3/library/asyncio-policy.html#asyncio.DefaultEventLoopPolicy

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/encode/httpx/issues/914#issuecomment-622297443, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAINSPTJI3OZEDU6VKCF2KLRPKBHTANCNFSM4MVRF2GQ .

Baysul commented 4 years ago

The workaround is to use the selector event loop like this:

import sys
import asyncio

if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.startswith('win'):
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

The following from the documentation will not resolve this issue (for some reason):

import asyncio
import selectors

selector = selectors.SelectSelector()
loop = asyncio.SelectorEventLoop(selector)
asyncio.set_event_loop(loop)

rmawatson commented 4 years ago

Thanks @Baysul.

This bug report looks like it is tracking the same issue.

tomchristie commented 4 years ago

Same as with #825 - Haven't looked into how awkward it is to do, but really I think we'd want to monkeypatch asyncio's socket unwrapping behaviour, so that we can reliably call .wait_closed(...) without it potentially hanging.

tomchristie commented 4 years ago

Can anyone on Windows, running Python 3.8+ confirm what currently happens when running this?...

import httpx
import asyncio

async def main():
    async with httpx.AsyncClient() as client:
        r = await client.get("https://login.microsoftonline.com/")
        print(r)

asyncio.run(main(), debug=True)

Trying to verify if we're still exposed to this issue. (I guess we are but 🤷‍♂️)

dalf commented 4 years ago

Using:

Windows 10 64bit
Python 3.8.5
httpx 0.14.1

I have a httpx._exceptions.ConnectTimeout even with timeout=60 or http2=True.

tomchristie commented 4 years ago

@dalf - Thanks!

Also uh... that's not at all what I was expecting / looking for. 🙃🙃🙃

Does that replicate on other URLs?...

"https://www.google.com/"
"https://www.example.org/"

Does removing the debug=True change anything?

Does it work okay in the sync version?...

import httpx

def main():
   with httpx.Client() as client:
        r = client.get("https://login.microsoftonline.com/")
        print(r)

main()

Or with trio?... (Use pip install trio)

import httpx
import trio

async def main():
    async with httpx.AsyncClient() as client:
        r = await client.get("https://login.microsoftonline.com/")
        print(r)

trio.run(main)

dalf commented 4 years ago

Maybe my report is irrelevant, I'm not familiar with the Python Windows environment:

import httpx
import requests

r_requests = requests.get('https://www.google.com')
r_httpx = httpx.get('https://www.google.com')

requests returns a normal response
httpx raises ConnectTimeout: _ssl.c:1106: The handshake operation timed out
the aiohttp client example works (on https://www.google.com )

tomchristie commented 4 years ago

🤨

That's pretty relevant. But just super surprising.

Are you able to double double check you've got fully up to date httpx and httpcore versions installed? Is anyone else able to confirm?

dalf commented 4 years ago

To do the test, I've installed Python from https://www.python.org/ftp/python/3.8.5/python-3.8.5.exe (it wasn't installed before), and just pip install httpx[http2]

pip freeze shows

httpcore==0.10.2
httpx==0.14.1

I think a test from someone else would be helpful.

j178 commented 4 years ago

I ran some tests:

Test against `https://github.com/`

Sync client works fine:

import httpx
import requests

resp = httpx.get('https://github.com/')
print('httpx', resp)

resp = requests.get('https://github.com/')
print('requests', resp)

results:

httpx <Response [200 OK]>
requests <Response [200]>

Async client, run with `asyncio`,

import httpx
import asyncio

async def main():
    async with httpx.AsyncClient() as client:
        r = await client.get("https://github.com/")
        print(r)

asyncio.run(main(), debug=True)

results:

<Response [200 OK]>
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000024774DEA0D0>
Traceback (most recent call last):
  File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "D:\Programs\Python38\lib\asyncio\base_events.py", line 719, in call_soon
    self._check_closed()
  File "D:\Programs\Python38\lib\asyncio\base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Async client, run with `trio`, works fine:

import httpx
import trio

async def main():
    async with httpx.AsyncClient() as client:
        r = await client.get("https://github.com/")
        print(r)

trio.run(main)

results:

<Response [200 OK]>

Test against `https://login.microsoftonline.com/`

Run with sync client

import httpx
import requests

resp = httpx.get('https://login.microsoftonline.com/')
print('httpx', resp)

resp = requests.get('https://login.microsoftonline.com/')
print('requests', resp)

results:

requests always succeed, httpx *sometimes* failed with `ConnectTimeout`

``` requests Traceback (most recent call last): File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 342, in map_exceptions yield File "e:\projects\pycharm\httpx\httpx\_client.py", line 798, in _send_single_request ) = transport.request( File "e:\projects\pycharm\httpcore\httpcore\_sync\connection_pool.py", line 188, in request response = connection.request( File "e:\projects\pycharm\httpcore\httpcore\_sync\connection.py", line 83, in request self.socket = self._open_socket(timeout) File "e:\projects\pycharm\httpcore\httpcore\_sync\connection.py", line 104, in _open_socket return self.backend.open_tcp_stream( File "e:\projects\pycharm\httpcore\httpcore\_backends\sync.py", line 144, in open_tcp_stream return SyncSocketStream(sock=sock) File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpcore\httpcore\_exceptions.py", line 12, in map_exceptions raise to_exc(exc) from None httpcore.ConnectTimeout: _ssl.c:1106: The handshake operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File ".\sync.py", line 12, in resp = httpx.get(url) File "e:\projects\pycharm\httpx\httpx\_api.py", line 174, in get return request( File "e:\projects\pycharm\httpx\httpx\_api.py", line 89, in request return client.request( File "e:\projects\pycharm\httpx\httpx\_client.py", line 673, in request return self.send( File "e:\projects\pycharm\httpx\httpx\_client.py", line 703, in send response = self._send_handling_redirects( File "e:\projects\pycharm\httpx\httpx\_client.py", line 732, in _send_handling_redirects response = self._send_handling_auth( File "e:\projects\pycharm\httpx\httpx\_client.py", line 769, in _send_handling_auth response = self._send_single_request(request, timeout) File "e:\projects\pycharm\httpx\httpx\_client.py", line 792, in _send_single_request ( File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 359, in map_exceptions raise mapped_exc(message, **kwargs) from exc # type: ignore httpx.ConnectTimeout: _ssl.c:1106: The handshake operation timed out ```

Running with `asyncio`

import httpx
import asyncio

async def main():
    async with httpx.AsyncClient() as client:
        r = await client.get("https://login.microsoftonline.com/")
        print(r)

asyncio.run(main(), debug=True)

results in three different types error(in an abritary way):

Error 1: Request succeed but with RuntimeError

``` Exception ignored in: Traceback (most recent call last): File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 116, in __del__ self.close() File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 108, in close self._loop.call_soon(self._call_connection_lost, None) File "D:\Programs\Python38\lib\asyncio\base_events.py", line 719, in call_soon self._check_closed() File "D:\Programs\Python38\lib\asyncio\base_events.py", line 508, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Exception ignored in: Traceback (most recent call last): File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 116, in __del__ self.close() File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 108, in close self._loop.call_soon(self._call_connection_lost, None) File "D:\Programs\Python38\lib\asyncio\base_events.py", line 719, in call_soon self._check_closed() File "D:\Programs\Python38\lib\asyncio\base_events.py", line 508, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed ```

Error 2: ConnectTimeout

``` Traceback (most recent call last): File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 342, in map_exceptions yield File "e:\projects\pycharm\httpx\httpx\_client.py", line 1402, in _send_single_request ) = await transport.request( File "e:\projects\pycharm\httpcore\httpcore\_async\connection_pool.py", line 188, in request response = await connection.request( File "e:\projects\pycharm\httpcore\httpcore\_async\connection.py", line 83, in request self.socket = await self._open_socket(timeout) File "e:\projects\pycharm\httpcore\httpcore\_async\connection.py", line 104, in _open_socket return await self.backend.open_tcp_stream( File "e:\projects\pycharm\httpcore\httpcore\_backends\auto.py", line 40, in open_tcp_stream return await self.backend.open_tcp_stream( File "e:\projects\pycharm\httpcore\httpcore\_backends\asyncio.py", line 240, in open_tcp_stream File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpcore\httpcore\_exceptions.py", line 12, in map_exceptions raise to_exc(exc) from None httpcore.ConnectTimeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File ".\async_test.py", line 13, in asyncio.run(main(), debug=True) File "D:\Programs\Python38\lib\asyncio\runners.py", line 43, in run return loop.run_until_complete(main) File "D:\Programs\Python38\lib\asyncio\base_events.py", line 616, in run_until_complete return future.result() File ".\async_test.py", line 10, in main r = await client.get("https://login.microsoftonline.com/") File "e:\projects\pycharm\httpx\httpx\_client.py", line 1441, in get return await self.request( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1274, in request response = await self.send( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1305, in send response = await self._send_handling_redirects( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1334, in _send_handling_redirects File "e:\projects\pycharm\httpx\httpx\_client.py", line 1371, in _send_handling_auth response = await self._send_single_request(request, timeout) File "e:\projects\pycharm\httpx\httpx\_client.py", line 1396, in _send_single_request ( File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 359, in map_exceptions raise mapped_exc(message, **kwargs) from exc # type: ignore httpx.ConnectTimeout Exception ignored in: Traceback (most recent call last): File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 116, in __del__ File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 108, in close File "D:\Programs\Python38\lib\asyncio\base_events.py", line 719, in call_soon File "D:\Programs\Python38\lib\asyncio\base_events.py", line 508, in _check_closed RuntimeError: Event loop is closed ```

Error 3: ReadTimeout

``` Traceback (most recent call last): File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 342, in map_exceptions yield File "e:\projects\pycharm\httpx\httpx\_client.py", line 1402, in _send_single_request ) = await transport.request( File "e:\projects\pycharm\httpcore\httpcore\_async\connection_pool.py", line 188, in request response = await connection.request( File "e:\projects\pycharm\httpcore\httpcore\_async\connection.py", line 96, in request return await self.connection.request(method, url, headers, stream, timeout) File "e:\projects\pycharm\httpcore\httpcore\_async\http11.py", line 73, in request ) = await self._receive_response(timeout) File "e:\projects\pycharm\httpcore\httpcore\_async\http11.py", line 130, in _receive_response event = await self._receive_event(timeout) File "e:\projects\pycharm\httpcore\httpcore\_async\http11.py", line 160, in _receive_event data = await self.socket.read(self.READ_NUM_BYTES, timeout) File "e:\projects\pycharm\httpcore\httpcore\_backends\asyncio.py", line 134, in read return await asyncio.wait_for( File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpcore\httpcore\_exceptions.py", line 12, in map_exceptions raise to_exc(exc) from None httpcore.ReadTimeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File ".\async_test.py", line 13, in asyncio.run(main(), debug=True) File "D:\Programs\Python38\lib\asyncio\runners.py", line 43, in run return loop.run_until_complete(main) File "D:\Programs\Python38\lib\asyncio\base_events.py", line 616, in run_until_complete return future.result() File ".\async_test.py", line 10, in main r = await client.get("https://login.microsoftonline.com/") File "e:\projects\pycharm\httpx\httpx\_client.py", line 1441, in get return await self.request( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1274, in request response = await self.send( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1305, in send response = await self._send_handling_redirects( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1334, in _send_handling_redirects response = await self._send_handling_auth( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1371, in _send_handling_auth response = await self._send_single_request(request, timeout) File "e:\projects\pycharm\httpx\httpx\_client.py", line 1396, in _send_single_request ( File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 359, in map_exceptions raise mapped_exc(message, **kwargs) from exc # type: ignore httpx.ReadTimeout Fatal error on SSL transport protocol: transport: <_ProactorSocketTransport fd=372 read=<_OverlappedFuture cancelled created at D:\Programs\Python38\lib\asyncio\windows_events.py:461>> Traceback (most recent call last): File "D:\Programs\Python38\lib\asyncio\sslproto.py", line 685, in _process_write_backlog self._transport.write(chunk) File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 359, in write self._loop_writing(data=bytes(data)) File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 395, in _loop_writing self._write_fut = self._loop._proactor.send(self._sock, data) AttributeError: 'NoneType' object has no attribute 'send' Exception ignored in: Traceback (most recent call last): File "D:\Programs\Python38\lib\asyncio\sslproto.py", line 322, in __del__ File "D:\Programs\Python38\lib\asyncio\sslproto.py", line 317, in close File "D:\Programs\Python38\lib\asyncio\sslproto.py", line 594, in _start_shutdown File "D:\Programs\Python38\lib\asyncio\sslproto.py", line 599, in _write_appdata File "D:\Programs\Python38\lib\asyncio\sslproto.py", line 707, in _process_write_backlog File "D:\Programs\Python38\lib\asyncio\sslproto.py", line 721, in _fatal_error File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 151, in _force_close File "D:\Programs\Python38\lib\asyncio\base_events.py", line 719, in call_soon File "D:\Programs\Python38\lib\asyncio\base_events.py", line 508, in _check_closed RuntimeError: Event loop is closed Exception ignored in: Traceback (most recent call last): File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 116, in __del__ File "D:\Programs\Python38\lib\asyncio\proactor_events.py", line 108, in close File "D:\Programs\Python38\lib\asyncio\base_events.py", line 719, in call_soon File "D:\Programs\Python38\lib\asyncio\base_events.py", line 508, in _check_closed RuntimeError: Event loop is closed ```

Run with trio

Sometimes succeed, sometimes failed with `ReadTimeout` or `ConnectTimeout`

ReadTimeout: ``` Traceback (most recent call last): File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 342, in map_exceptions yield File "e:\projects\pycharm\httpx\httpx\_client.py", line 1402, in _send_single_request ) = await transport.request( File "e:\projects\pycharm\httpcore\httpcore\_async\connection_pool.py", line 188, in request response = await connection.request( File "e:\projects\pycharm\httpcore\httpcore\_async\connection.py", line 96, in request return await self.connection.request(method, url, headers, stream, timeout) File "e:\projects\pycharm\httpcore\httpcore\_async\http11.py", line 73, in request ) = await self._receive_response(timeout) File "e:\projects\pycharm\httpcore\httpcore\_async\http11.py", line 130, in _receive_response event = await self._receive_event(timeout) File "e:\projects\pycharm\httpcore\httpcore\_async\http11.py", line 160, in _receive_event data = await self.socket.read(self.READ_NUM_BYTES, timeout) File "e:\projects\pycharm\httpcore\httpcore\_backends\trio.py", line 63, in read return await self.stream.receive_some(max_bytes=n) File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpcore\httpcore\_exceptions.py", line 12, in map_exceptions raise to_exc(exc) from None httpcore.ReadTimeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File ".\async_test-trio.py", line 12, in trio.run(main) File "D:\Programs\Python38\lib\site-packages\trio\_core\_run.py", line 1896, in run raise runner.main_task_outcome.error File ".\async_test-trio.py", line 9, in main r = await client.get("https://login.microsoftonline.com/") File "e:\projects\pycharm\httpx\httpx\_client.py", line 1441, in get return await self.request( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1274, in request response = await self.send( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1305, in send response = await self._send_handling_redirects( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1334, in _send_handling_redirects response = await self._send_handling_auth( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1371, in _send_handling_auth response = await self._send_single_request(request, timeout) File "e:\projects\pycharm\httpx\httpx\_client.py", line 1396, in _send_single_request ( File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 359, in map_exceptions raise mapped_exc(message, **kwargs) from exc # type: ignore httpx.ReadTimeout ``` ConnectTimeout: ``` Traceback (most recent call last): File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 342, in map_exceptions yield File "e:\projects\pycharm\httpx\httpx\_client.py", line 1402, in _send_single_request ) = await transport.request( File "e:\projects\pycharm\httpcore\httpcore\_async\connection_pool.py", line 188, in request response = await connection.request( File "e:\projects\pycharm\httpcore\httpcore\_async\connection.py", line 83, in request self.socket = await self._open_socket(timeout) File "e:\projects\pycharm\httpcore\httpcore\_async\connection.py", line 104, in _open_socket return await self.backend.open_tcp_stream( File "e:\projects\pycharm\httpcore\httpcore\_backends\auto.py", line 40, in open_tcp_stream return await self.backend.open_tcp_stream( File "e:\projects\pycharm\httpcore\httpcore\_backends\trio.py", line 168, in open_tcp_stream return SocketStream(stream=stream) File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpcore\httpcore\_exceptions.py", line 12, in map_exceptions raise to_exc(exc) from None httpcore.ConnectTimeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File ".\async_test-trio.py", line 12, in trio.run(main) File "D:\Programs\Python38\lib\site-packages\trio\_core\_run.py", line 1896, in run raise runner.main_task_outcome.error File ".\async_test-trio.py", line 9, in main r = await client.get("https://login.microsoftonline.com/") File "e:\projects\pycharm\httpx\httpx\_client.py", line 1441, in get return await self.request( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1274, in request response = await self.send( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1305, in send response = await self._send_handling_redirects( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1334, in _send_handling_redirects response = await self._send_handling_auth( File "e:\projects\pycharm\httpx\httpx\_client.py", line 1371, in _send_handling_auth response = await self._send_single_request(request, timeout) File "e:\projects\pycharm\httpx\httpx\_client.py", line 1396, in _send_single_request ( File "D:\Programs\Python38\lib\contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "e:\projects\pycharm\httpx\httpx\_exceptions.py", line 359, in map_exceptions raise mapped_exc(message, **kwargs) from exc # type: ignore httpx.ConnectTimeout ```

Using:

Windows 10 64bit
Python 3.8.5
httpx latest master branch
httpcore latest master branch

kpostekk commented 3 years ago

I used to have this issue too. But surprisingly, this issue no longer occurs in any of my projects. I have tested it few times on Python 3.8.6 and Python 3.9.1. I think this issue can be closed for now.

Versions:

httpx==0.16.1
  - certifi [required: Any, installed: 2020.12.5]
  - httpcore [required: ==0.12.*, installed: 0.12.3]
    - h11 [required: ==0.*, installed: 0.12.0]
    - sniffio [required: ==1.*, installed: 1.2.0]
  - rfc3986 [required: >=1.3,<2, installed: 1.4.0]
  - sniffio [required: Any, installed: 1.2.0]

Edit: Os version

Version         Windows 10 Pro
Version         20H2
Compilation 19042.746
Features        Windows Feature Experience Pack 120.2212.551.0

florimondmanca commented 3 years ago

It might be related to some recent improvements in HTTPCore (especially some related to socket management).

Is anyone else able to confirm whether this seems resolved using the versions above?

kpostekk commented 3 years ago

It might be related to some recent improvements in HTTPCore

I have ran some tests. Error raises on these versions of HTTPCore: 0.12.0, 0.12.1 and 0.12.2

I can do a pull request with fixed requirements.

florimondmanca commented 3 years ago

Coolio. The likely series of events is that prior to 0.12.2, we were scheduling the closing of sockets on the event loop but not awaiting, which might have caused closing to happen after event loop closure on Windows for some reason. On 0.12.3 we are now properly waiting for sockets to close whenever possible.

Thanks for the PR proposition but I guess we can just have existing users upgrade rather than update our pin and issue a new release of HTTPX.

wolph commented 3 years ago

While certainly not a fix, this has worked for me in many cases:

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.run_until_complete(asyncio.sleep(1))
loop.close()

florimondmanca commented 3 years ago

@WoLpH 👋 Are you able to confirm if HTTPCore 0.12.3 makes the issue go away if you remove that sleep(1) workaround?

wolph commented 3 years ago

Yes, it solves the issue :)

I still think the workaround can be useful for people though. There are many scenarios where this is an issue from my experience

florimondmanca commented 3 years ago

Nice. I'll close this off now then. Thanks all!

nkrsic commented 2 years ago

Also: May well be specific to Window's Proactor event loop.

I am experiencing this as well for anyone reading this in posterity, Windows 10 getting the event loop already closed error. The code terminates normally on Linux.

mbwmbw1337 commented 2 years ago

All -

I am seeing the same issue on Windows. The time.sleep(1) has fixed the problem thank you @rmawatson - this has been driving me crazy.

Mennaruuk commented 2 years ago

When I tried workarounds such as the one here, what happened was that at the end of the loop, Python exited. And that was it. Even when I have code after the loop, it didn't care about it and just terminated everything. Unfortunately, the time.sleep(10) trick didn't work, and neither did this solution.

Are there any other workarounds? Thank you.

https://user-images.githubusercontent.com/52135169/152754459-16ca58f5-e602-4ae3-bbf0-692d4ee53c29.mp4

tomchristie commented 2 years ago

@Mennaruuk (or anyone else seeing this) - Let's take this from the start.

What's the simplest possible script you can show us, that demonstrates the issue.
What's the exact behaviour/traceback you're seeing?
Which version of Python are you using?
Which versions of httpx and httpcore do you have installed?

Mennaruuk commented 2 years ago

Dear Tom,

The script

Here's the simplest script that I could reduce the original script to while still allowing for the error to be reproduced.

Sample command to run: script.py -u jack -from 20170101 -to 20170102

From trial and error, I learned the following:

The problem seems to involve argparse. In the script above, there are three arguments (-u, -from, and -to). When I remove -from and -to and keep just -u, the script works. In short, multiple arguments in argparse lead to the script failing. Example script. Sample command to run: script.py -u jack -from 20170101 -to 20170102
There is one exception to the prior sentence. You see, I have at the end of the original script some if/elif/else statements. If I keep the elif, the script fails. But if I remove the elif part, the script works. Example script. Sample command to run: script.py -u jack

The behavior

https://user-images.githubusercontent.com/52135169/152915595-d9070c6e-871d-4a9d-96b8-05e4f3fa9013.mp4

Script suddenly terminates after the async process is done. Despite the presence of input functions right after the async function, Python terminates before the input functions are given a chance to run. See screenshot below for the input functions code following the async code.

Versions

Python: 3.10.0 httpx: 0.22.0 httpcore: 0.14.7

If you need any more information, please let me know. I'm happy to provide you with any information to the best of my ability. Here's the not-so-simple script that I initially worked from if it's of help.

pprovart commented 2 years ago

Windows 10 Python 3.10.2 NOTE: Problem occured before and after i installed httpx and httpcore httpcore 0.14.7 httpx 0.22.0

Code gets the required data from BSCSCAN and then i get the Runtime Error. The time.sleep(1) command didnt work.

PYTHON import asyncio from bscscan import BscScan

YOUR_API_KEY = "REDACTED" CON_ADDR = "0xa38898a4ae982cb0131104a6746f77fa0da57aaa" WAL_ADDR = "0x411709fd0cdd5f06ff6ef45b48aa0cb9842f01fc"

async def main(): async with BscScan(YOUR_API_KEY) as bsc: print(await bsc.get_bnb_balance(address="0xa38898a4ae982cb0131104a6746f77fa0da57aaa")

if name == "main": asyncio.run(main())

OUTPUT

aabmets commented 2 years ago

I'm having the same issue on Windows 10. It does not matter what library I'm using. If the loop is an ProactorEventLoop, then it is guaranteed that it will cause an exception if there have been any network requests made which use SSL or sessions.

tomchristie commented 2 years ago

Okay - could you show me that absolute simplest possible example that reproduces this.

@pprovart's example here is pretty good - does it still replicate if you simplify it even further?...

import asyncio
import httpx

async def main():
    async with httpx.Client() as client:
        r = await client.get("https://www.example.com")
        print(r)

if name == "main":
    asyncio.run(main())

Also, does it replicate with both http and https URLs?

Mennaruuk commented 2 years ago

Nope, it works pretty well for me. I get <Response [301 Moved Permanently]> with and without HTTPS. I looked at this proposed solution, and I modified it as the following. It worked for me, not sure if it works for others. but I stopped getting the RuntimeError:

import platform

if platform.system() == 'Windows':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

tomchristie commented 2 years ago

@Mennaruuk - Ah yup. Just looked though this thread and reminded myself of the comment here.

This is a bug in Python's stdlib with Windows. See: https://bugs.python.org/issue39232

One option here might be to check if the ProactorEventLoop is the default, and warn if it is.

graingert commented 2 years ago

when using async with httpx.AsyncClient() as client the transport should be closed correctly by anyio and not trigger the __del__ so this looks like an anyio bug

graingert commented 2 years ago

@pprovart is using aiohttp and not httpx https://github.com/pcko1/bscscan-python/blob/fcfc30cb7d1a4ff2a4f2de30606f35698675cbc7/bscscan/core/async_client.py#L32

tomchristie commented 1 year ago

This bug in the Python stdlib appears to have been fixed now in 3.11 (and possibly backported to 3.10?)

https://github.com/python/cpython/issues/83413

Are any windows users here able to attempt replicating the original issue on the latest patch versions of Python 3.9, 3.10 and 3.12?

mbwmbw1337 commented 1 year ago

This bug in the Python stdlib appears to have been fixed now in 3.11 (and possibly backported to 3.10?)

python/cpython#83413

Are any windows users here able to attempt replicating the original issue on the latest patch versions of Python 3.9, 3.10 and 3.12?

I can test later today.

FANGOD commented 1 year ago

This problem also occurs on linux.

Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7f85c758ce50>
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/base_subprocess.py", line 126, in __del__
    self.close()
  File "/usr/lib/python3.8/asyncio/base_subprocess.py", line 104, in close
    proto.pipe.close()
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 536, in close
    self._close(None)
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 560, in _close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 711, in call_soon
    self._check_closed()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 504, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

kpostekk commented 1 year ago

This problem also occurs on linux.

Have you considered switching to recent version of python?

encode / httpx