search

ESGF / esgf-download

ESGF data transfer and replication tool
https://esgf.github.io/esgf-download/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Connection pool was closed while 1 HTTP requests/repsonses were still in-flight. #12

Closed AtefBN closed 1 year ago

AtefBN commented 1 year ago

Not a very clear explicit error message but this has been blocking a set of downloads consistently over a few days:

During handling of the above exception, another exception occurred:

[2023-06-14 12:26:07] DEBUG root Locals: { 'self': <httpcore.AsyncConnectionPool object at 0x7fc6c6e3aad0>, 'requests_still_in_flight': 1, 'connection': <AsyncHTTPConnection ['https://esgf.ceda.ac.uk:443', HTTP/1.1, CLOSED, Request Count: 1]> }

[2023-06-14 12:26:07] ERROR root

Traceback (most recent call last): File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 109, in connect_tcp stream: anyio.abc.ByteStream = await anyio.connect_tcp( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 175, in connect_tcp async with create_task_group() as tg: File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 574, in aexit raise exceptions[0] File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 636, in task_done exc = _task.exception() ^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 127, in try_connect stream = await asynclib.connect_tcp(remote_host, remote_port, local_address) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1518, in connect_tcp await get_running_loop().create_connection(StreamProtocol, host, port, File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection sock = await self._connect_sock( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock await self.sock_connect(sock, address) File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect return await fut ^^^^^^^^^ asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 108, in connect_tcp with anyio.fail_after(timeout): File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/anyio/_core/_tasks.py", line 103, in exit raise TimeoutError TimeoutError During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions yield File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request raise exc File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request raise exc File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request stream = await self._connect(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/_async/connection.py", line 111, in _connect stream = await self._network_backend.connect_tcp(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp return await self._backend.connect_tcp( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/backends/asyncio.py", line 107, in connect_tcp with map_exceptions(exc_map): File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/contextlib.py", line 155, in exit self.gen.throw(typ, value, traceback) File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions raise to_exc(exc) httpcore.ConnectTimeout

svenrdz commented 1 year ago

This is fixed since https://github.com/ESGF/esgf-download/commit/8e0de5679d8a62526c899e20d9671a2b5a6f5962