Open jakirkham opened 5 months ago
cc @raydouglass @mmccarty (for vis)
@dholth could you please help us look at this?
Looks like this issue started last week: https://github.com/conda/infrastructure/issues/899#issuecomment-2026232053 (right before a company break)
Just trying to rule out possible issues; we encounter network issues when switching from mamba
to conda
invocations as well.
https://github.com/rapidsai/docker/actions/runs/8528278564/job/23361587494?pr=650#step:11:236
#11 60.18 CondaHTTPError: HTTP 520 CONNECTION FAILED for url <https://conda.anaconda.org/dask/label/dev/noarch/repodata.json>
https://github.com/rapidsai/docker/actions/runs/8528278564/job/23361586014?pr=650#step:9:1193
#12 156.8 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='conda.anaconda.org', port=443): Read timed out. (read timeout=60.0)")': /rapidsai-nightly/linux-64/dask-cudf-24.04.00a586-cuda11_py311_240402_g35f818b3e4_586.tar.bz2
#12 343.0 CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/rapidsai-nightly/linux-64/dask-cudf-24.04.00a586-cuda11_py311_240402_g35f818b3e4_586.tar.bz2>
Also I cannot say with 100% confidence, but so far all of the errors I've seen reviewing logs from the past week involved the rapidsai-nightly
or dask/label/dev
channels, but could be a coincidence since we download a lot of large packages from rapidsai-nightly
and so the odds of a network error is higher.
I have not checked every single error though.
cc @mariusvniekerk (for awareness)
Also saw this with rapidsai
in this CI job
error libmamba ZSTD decompression error: Unknown frame descriptor
Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/rapidsai/noarch/repodata.json.zst]
Failure writing output to destination, passed 689 returned 690
Have also see this with cf-staging
. Snippet below from this GHA job:
E binstar_client.errors.ServerError: ('?: Undefined error ([GET] https://api.anaconda.org/dist/cf-staging/blah-2696ff/2024.04.03.03.11.49/noarch%2Fblah-2696ff-2024.04.03.03.11.49-py_0.tar.bz2 -> 524)', 524)
Just noting that we're investigating this still
Hey everyone, quick info from the infrastructure side of Anaconda: We're on this and have found an issue in the underlying infrastructure that is likely to cause this. We're going to implement a fix for this suspected cause in the next few hours.
I'll let you know when this has been rolled out.
We have rolled out the configuration changes that should fix these timeouts. Please let us know if you continue to see these issues.
@jezdez & @morremeyer Thanks for the update!
We were consistently encountering the networking errors during one of our workflows over the past week+.
I triggered a rerun and it was able to successfully download all of the conda packages, so seems like things are improved!
We have rolled out the configuration changes that should fix these timeouts. Please let us know if you continue to see these issues.
I am continuing seeing these kind of problems in the robotology
or robostack-staging
channel (posting here as I guess the problem is similar, if you prefer that I open a different issue just let me know). Example CI failure:
Restarting the CI typically solve this issues.
Are others still seeing this? If so, a fresh reproducer would help
If not, would suggest we close to clean up the issue tracker and focus efforts on current issues
Based on https://github.com/robotology/icub-models/actions/workflows/cxx-ci.yml it sees to me that it is still happening, at a rough rate of 1 in ~20 builds, but much less frequently that happened in early april, when the failure rate was 1 in ~3/4 builds.
Ok for me in closing if keeping the issue open is not useful.
Current examples seem worthy of discussion
No strong feelings as to whether that stays in this issue or is moved to a new one
Checklist
What happened?
Conda Info
Conda Config
Conda list
Additional Context
Here is the full log