search

dandi / dandisets

730 Dandisets, 807.1 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

sync of 000026 failed due to ssl.SSLZeroReturnError #341

Closed yarikoptic closed 1 year ago

yarikoptic commented 1 year ago

we might need to add a retry to robustify

2023-04-06T16:50:26-0400 [WARNING ] backups2datalad: Retrying HEAD request to https://dandiarchive.s3.amazonaws.com/blobs/5ba/587/5ba58725-c70e-4b45-ad2b-f19731b940fc in 1.015132 seconds as it raised ConnectTimeout:
2023-04-06T16:50:26-0400 [WARNING ] backups2datalad: Retrying HEAD request to https://dandiarchive.s3.amazonaws.com/blobs/166/b6e/166b6ee4-59fb-4c87-ba3d-ce7012b2f06f in 1.015866 seconds as it raised ConnectTimeout:
2023-04-06T16:58:44-0400 [WARNING ] backups2datalad: Retrying HEAD request to https://dandiarchive.s3.amazonaws.com/blobs/818/ffc/818ffcc9-32de-43ad-975a-e2dc81d87ee7 in 0.972968 seconds as it raised RemoteProtocolError: Server disconnected without sending a response.
2023-04-06T17:00:17-0400 [ERROR   ] backups2datalad: Job failed on input <Dandiset 000026/draft>:
Traceback (most recent call last):
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 471, in async_assets
    async with await open_git_annex(
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 262, in process_asset
    bucket_url = await self.get_file_bucket_url(asset)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 352, in get_file_bucket_url
    r = await arequest(
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 126, in arequest
    r = await client.request(method, url, follow_redirects=True, **kwargs)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpx/_client.py", line 1506, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpx/_client.py", line 1593, in send
    response = await self._send_handling_auth(
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpx/_client.py", line 1621, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpx/_client.py", line 1658, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpx/_client.py", line 1695, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/_async/connection.py", line 150, in _connect
    stream = await stream.start_tls(**kwargs)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 72, in start_tls
    raise exc
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 63, in start_tls
    ssl_stream = await anyio.streams.tls.TLSStream.wrap(
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/streams/tls.py", line 122, in wrap
    await wrapper._call_sslobject_method(ssl_object.do_handshake)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/streams/tls.py", line 130, in _call_sslobject_method
    result = func(*args)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/ssl.py", line 974, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 171, in dowork
    outp = await func(inp)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 141, in update_dandiset
    changed = await self.sync_dataset(dandiset, ds, dmanager)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 186, in sync_dataset
    await syncer.sync_assets(error_on_change)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets
    self.report = await async_assets(
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 471, in async_assets
    async with await open_git_annex(
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpx/_client.py", line 1975, in __aexit__
    await self._transport.__aexit__(exc_type, exc_value, traceback)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpx/_transports/default.py", line 332, in __aexit__
    await self._pool.__aexit__(exc_type, exc_value, traceback)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 326, in __aexit__
    await self.aclose()
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 312, in aclose
    raise RuntimeError(
RuntimeError: The connection pool was closed while 57 HTTP requests/responses were still in-flight.
2023-04-06T17:00:17-0400 [ERROR   ] backups2datalad: An error occurred:
Traceback (most recent call last):
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 111, in wrapped
    await f(datasetter, *args, **kwargs)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 213, in update_from_backup
    await datasetter.update_from_backup(dandisets, exclude=exclude)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 95, in update_from_backup
    raise RuntimeError(
RuntimeError: Backups for 1 Dandiset failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2023.04.06.20.43.00Z.log
yarikoptic commented 1 year ago

Let's add retry on SSL error @jwodder - from above it seems we are not continuing to retry if SSL error happens