Slow CDN mirroring - Githubissues

jakirkham commented 4 months ago

It looks like it is taking over 1hr for some packages to mirror. For example:

Do we know what might be causing this?

jakirkham commented 4 months ago

Here's another example:

stefanandres commented 4 months ago

We see some errors in the channel-clone VMs, we are checking if this is related

jakirkham commented 4 months ago

Thanks Stefan! 🙏

Interested to learn what you discover 🙂

jakirkham commented 4 months ago

Adding a more recent example in case it is helpful

jakirkham commented 4 months ago

This one is approaching ~2.5hrs

Idk if there is something more going on with this case (hence mentioning it explicitly)

Edit: Seeing the same issue with this one

h-vetinari commented 4 months ago

Another example after John referred me to this issue: 2+ hours after marking some packages as broken in https://github.com/conda-forge/admin-requests/commit/164a4271b51ee47fdcfdd52905e0137fbdf21003, packages are still not removed from the CDN (it's difficult to do quickfixes to wide-ranging breaks like this one if the turnaround is that long).

>mamba repoquery search -c conda-forge gcc_linux-aarch64=12.3.0=*_2 -p linux-64

Executing the query gcc_linux-aarch64=12.3.0=*_2

conda-forge/noarch                                  13.9MB @   3.8MB/s  3.6s
conda-forge/linux-64                                33.0MB @   4.2MB/s  7.8s

 Name              Version Build                   Channel     Subdir
------------------------------------------------------------------------
 gcc_linux-aarch64 12.3.0  h490a0b6_2 (+ 1 builds) conda-forge linux-64       # should not contain h490a0b6_2

>python
Python 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:40:50) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.Timestamp.now("utc")
Timestamp('2024-03-12 05:43:31.911276+0000', tz='UTC')                        # marked broken >2h before

dholth commented 4 months ago

We see some files generated by anaconda.org's dynamic repodata like linux-aarch64/awscli-1.29.39-py38he37f277_0.conda that can't be downloaded, and others like noarch/strawberry-graphql-with-asgi-0.162.0-pyhd8ed1ab_0.conda that cannot be re-indexed as part of the CDN process.

jakirkham commented 3 months ago

Interesting, thanks Daniel! 🙏

Is there something in regards to those packages specifically that looks relevant?

Or did something unrelated to those packages occur (like a network outage or running out of disk space)?

jakirkham commented 3 months ago

Were we able to determine the cause here?

dholth commented 3 months ago

We were able to find packages that have repodata but no downloadable archive missing-packages.txt

We were also able to fix a bug that is more likely to be the cause, where we would have trouble re-downloading a package in the CDN process if the first clone failed. We were not able to find the precise cause.

jakirkham commented 3 months ago

Gotcha thanks for the update Daniel! 🙏

Looked at the first one on the list, linux-aarch64/awscli-1.29.39-py38he37f277_0.conda. It looks like other OS and arch combinations for 1.29.39 had validation issues ( https://github.com/conda-forge/awscli-feedstock/issues/765#issuecomment-1702350487 ), but linux-aarch64 was not one of these. These eventually this got sorted out. Can see a successful build and upload for this package. Here is the attached log. Though agree downloading this package doesn't work. So not sure what happened here. Maybe it should be marked broken (unclear whether that helps this issue)

The second package, noarch/boto3-stubs-lite-1.26.89-pyhd8ed1ab_0.conda, had a validation error with that package specifically ( https://github.com/conda-forge/boto3-stubs-lite-feedstock/issues/120#issuecomment-1465186684 ). Same with the third package, noarch/ca-policy-lcg-1.119-hd8ed1ab_0.conda, ( https://github.com/conda-forge/ca-policy-lcg-feedstock/issues/20#issue-1627340874 ). At least for these two cases, the conda-forge validation service notes the packages were not copied. However there do appear to be packages with 0 downloads for both boto3-stubs-lite and ca-policy-lcg. Visiting or downloading either results in a 404. Perhaps these should be marked broken as well

Looking at the latter two cases, do not see them in https://conda.anaconda.org/conda-forge/noarch/repodata.json.bz2. Nor do they show up in conda-forge-repodata-patches. So am wondering how these are added to the repodata when no package was copied

Idk if we can have aborted copy with the conda-forge validation service that might generate these issues, but that seems like one question that comes out of this

cc @beckermr (in case I'm missing anything here)

dholth commented 3 months ago

We download the dynamic anaconda.org repodata.json before creating the CDN version.

jakirkham commented 3 months ago

Can you please remind me which URL that lives under?

dholth commented 3 months ago

https://conda.anaconda.org/conda-forge/noarch/repodata-clone.json e.g.

h-vetinari commented 3 months ago

Just saw a particularly severe case today (still nothing after 3.5h for linux-64, 2h for win-64):

dholth commented 3 months ago

The logs show that the CDN clone process downloaded that archive from https://conda-web.anaconda.org/conda-forge/linux-64/clangdev-18.1.2-default_h127d8a8_0.conda, had a bad archive at 2024-03-21T02:29:30 and was able to get a good archive at 2024-03-21T06:12

h-vetinari commented 3 months ago

Not that I pretend to understand the cloning mechanism (or the reasons why it might fail), but would it make sense to have a shorter retry loop for failed clones? Like try again immediately after, or after X minutes delay?

dholth commented 3 months ago

It does retry frequently, there may be an intermediate cache issue.

beckermr commented 3 months ago

The CDN appears to be down again.

jaimergp commented 3 months ago

Approaching the 500min mark 😬 Should that metric be part of https://anaconda.statuspage.io/?

dholth commented 3 months ago

We've addressed a disk-full issue.

jakirkham commented 3 months ago

Just ran into a a network issue:

conda.CondaMultiError: ('Connection broken: IncompleteRead(199522674 bytes read, 79146888 more expected)', IncompleteRead(199522674 bytes read, 79146888 more expected))
('Connection broken: IncompleteRead(199522674 bytes read, 79146888 more expected)', IncompleteRead(199522674 bytes read, 79146888 more expected))

Wondering if this is related

jezdez commented 3 months ago

Approaching the 500min mark 😬 Should that metric be part of anaconda.statuspage.io?

Yes, I think we should start tracking this publicly somehow

beckermr commented 3 months ago

CDN is at 37 minutes.

dholth commented 3 months ago

Should be resolved.

jaimergp commented 3 months ago

Last sync was done almost 10h ago now.

jjerphan commented 3 months ago

I confirmed it was not updated for 24 hours.

jakirkham commented 3 months ago

Looking at this case, mirroring has not completed after ~40mins

jakirkham commented 3 months ago

The linux-aarch64 (highlighted) and win-64 package above it are still mirroring. Approaching the ~1.5hr mark

jakirkham commented 3 months ago

Regarding cuda-tools, the linux-aarch64 package is now available on CDN

However the win-64 package is not. It has been ~3.25hrs since it was uploaded

% conda search 'conda-forge:cuda-tools[subdir=win-64]=12.4.1'
Loading channels: done
No match found for: cuda-tools=12.4.1[subdir=win-64]. Search: *cuda-tools*=12.4.1[subdir=win-64]

PackagesNotFoundError: The following packages are not available from current channels:

  - cuda-tools=12.4.1[subdir=win-64]

Current channels:

  - https://conda.anaconda.org/conda-forge/win-64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

dholth commented 3 months ago

CONDA_DEBUG=1 conda search 'conda-forge::cuda-tools[subdir=win-64]=12.4.1' --json

jakirkham commented 3 months ago

Thanks Daniel! 🙏

Debug info:

conda info

``` active environment : base active env location : /Users/jkirkham/miniforge shell level : 1 user config file : /Users/jkirkham/.condarc populated config files : /Users/jkirkham/miniforge/.condarc /Users/jkirkham/.condarc conda version : 24.3.0 conda-build version : 24.3.0 python version : 3.10.13.final.0 solver : libmamba (default) virtual packages : __archspec=1=m1 __conda=24.3.0=0 __osx=14.4.1=0 __unix=0=0 base environment : /Users/jkirkham/miniforge (writable) conda av data dir : /Users/jkirkham/miniforge/etc/conda conda av metadata url : None channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64 https://conda.anaconda.org/conda-forge/noarch package cache : /Users/jkirkham/miniforge/pkgs /Users/jkirkham/.conda/pkgs envs directories : /Users/jkirkham/miniforge/envs /Users/jkirkham/.conda/envs platform : osx-arm64 user-agent : conda/24.3.0 requests/2.31.0 CPython/3.10.13 Darwin/23.4.0 OSX/14.4.1 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8 UID:GID : 502:20 netrc file : None offline mode : False ```

conda config --show-sources

``` ==> /Users/jkirkham/miniforge/.condarc <== channels: - conda-forge ==> /Users/jkirkham/.condarc <== channels: - conda-forge shortcuts: False solver: libmamba ```

conda list --show-channel-urls

``` # packages in environment at /Users/jkirkham/miniforge: # # Name Version Build Channel anaconda-client 1.12.3 pyhd8ed1ab_0 conda-forge anaconda-project 0.11.1 pyhd8ed1ab_0 conda-forge annotated-types 0.6.0 pyhd8ed1ab_0 conda-forge anyio 3.7.1 pyhd8ed1ab_0 conda-forge appnope 0.1.4 pyhd8ed1ab_0 conda-forge archspec 0.2.3 pyhd8ed1ab_0 conda-forge asttokens 2.4.1 pyhd8ed1ab_0 conda-forge attrs 23.2.0 pyh71513ae_0 conda-forge azure-core 1.30.1 pyhd8ed1ab_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 2.0.0 pyhd8ed1ab_0 conda-forge backports.strenum 1.3.1 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.12.3 pyha770c72_0 conda-forge blinker 1.7.0 pyhd8ed1ab_0 conda-forge boltons 24.0.0 pyhd8ed1ab_0 conda-forge boolean.py 4.0 pyhd8ed1ab_0 conda-forge brotli 1.1.0 hb547adb_1 conda-forge brotli-bin 1.1.0 hb547adb_1 conda-forge brotli-python 1.1.0 py310h1253130_1 conda-forge brotlipy 0.7.0 py310h2aa6e3c_1006 conda-forge bzip2 1.0.8 h93a5062_5 conda-forge c-ares 1.28.1 h93a5062_0 conda-forge ca-certificates 2024.2.2 hf0a4a13_0 conda-forge cctools 986 h4c9edd9_0 conda-forge cctools_osx-arm64 986 hd11630f_0 conda-forge certifi 2024.2.2 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py310hdcd7c05_0 conda-forge chardet 5.2.0 py310hbe9552e_1 conda-forge charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge cirun 0.30 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge clyent 1.2.2 pyhd8ed1ab_2 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge conda 24.3.0 py310hbe9552e_0 conda-forge conda-build 24.3.0 py310hbe9552e_1 conda-forge conda-forge-pinning 2024.04.09.19.22.01 hd8ed1ab_0 conda-forge conda-index 0.4.0 pyhd8ed1ab_0 conda-forge conda-libmamba-solver 24.1.0 pyhd8ed1ab_0 conda-forge conda-pack 0.7.1 pyhd8ed1ab_0 conda-forge conda-package-handling 2.2.0 pyh38be061_0 conda-forge conda-package-streaming 0.9.0 pyhd8ed1ab_0 conda-forge conda-smithy 3.34.1 pyhd8ed1ab_0 conda-forge conda-souschef 2.2.3 pyhd8ed1ab_0 conda-forge cryptography 42.0.5 py310h0e6f4b3_0 conda-forge curl 8.7.1 h2d989ff_0 conda-forge dataclasses 0.8 pyhc8e2a94_3 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge deprecated 1.2.14 pyh1a96a4e_0 conda-forge distlib 0.3.8 pyhd8ed1ab_0 conda-forge distro 1.9.0 pyhd8ed1ab_0 conda-forge editables 0.5 pyhd8ed1ab_0 conda-forge exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge expat 2.6.2 hebf3989_0 conda-forge filelock 3.13.4 pyhd8ed1ab_0 conda-forge fmt 10.2.1 h2ffa867_0 conda-forge freetype 2.12.1 hadb7bae_2 conda-forge gettext 0.22.5 h8fbad5d_2 conda-forge gettext-tools 0.22.5 h8fbad5d_2 conda-forge git 2.44.0 pl5321h015987d_0 conda-forge gitdb 4.0.11 pyhd8ed1ab_0 conda-forge gitpython 3.1.43 pyhd8ed1ab_0 conda-forge glob2 0.7 py_0 conda-forge gmp 6.3.0 hebf3989_1 conda-forge grayskull 2.2.2.dev1+gac11dca pypi_0 pypi h11 0.14.0 pyhd8ed1ab_0 conda-forge h2 4.1.0 pyhd8ed1ab_0 conda-forge hatch 1.9.4 pyhd8ed1ab_0 conda-forge hatchling 1.21.1 pyhd8ed1ab_0 conda-forge hpack 4.0.0 pyh9f0ad1d_0 conda-forge httpcore 1.0.5 pyhd8ed1ab_0 conda-forge httpx 0.27.0 pyhd8ed1ab_0 conda-forge hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge hyperlink 21.0.0 pyhd3deb0d_0 conda-forge icu 73.2 hc8870d7_0 conda-forge idna 3.6 pyhd8ed1ab_0 conda-forge importlib-metadata 7.1.0 pyha770c72_0 conda-forge importlib_metadata 7.1.0 hd8ed1ab_0 conda-forge importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge ipython 8.22.2 pyh707e725_0 conda-forge isodate 0.6.1 pyhd8ed1ab_0 conda-forge jaraco.classes 3.4.0 pyhd8ed1ab_0 conda-forge jaraco.context 4.3.0 pyhd8ed1ab_0 conda-forge jaraco.functools 4.0.0 pyhd8ed1ab_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jinja2 3.1.3 pyhd8ed1ab_0 conda-forge joblib 1.4.0 pyhd8ed1ab_0 conda-forge json5 0.9.24 pyhd8ed1ab_0 conda-forge jsonpatch 1.33 pyhd8ed1ab_0 conda-forge jsonpointer 2.4 py310hbe9552e_3 conda-forge jsonschema 4.21.1 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2023.12.1 pyhd8ed1ab_0 conda-forge jupyter_core 5.7.2 py310hbe9552e_0 conda-forge keyring 25.1.0 pyh534df25_0 conda-forge krb5 1.21.2 h92f50d5_0 conda-forge lcms2 2.16 ha0e7c42_0 conda-forge ld64 711 h4c6efb1_0 conda-forge ld64_osx-arm64 711 h5e7191b_0 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libarchive 3.7.2 hcacb583_1 conda-forge libasprintf 0.22.5 h8fbad5d_2 conda-forge libasprintf-devel 0.22.5 h8fbad5d_2 conda-forge libblas 3.9.0 17_osxarm64_openblas conda-forge libbrotlicommon 1.1.0 hb547adb_1 conda-forge libbrotlidec 1.1.0 hb547adb_1 conda-forge libbrotlienc 1.1.0 hb547adb_1 conda-forge libcblas 3.9.0 17_osxarm64_openblas conda-forge libcurl 8.7.1 h2d989ff_0 conda-forge libcxx 16.0.6 h4653b0c_0 conda-forge libdeflate 1.20 h93a5062_0 conda-forge libedit 3.1.20191231 hc8eb9b7_2 conda-forge libev 4.33 h93a5062_2 conda-forge libexpat 2.6.2 hebf3989_0 conda-forge libffi 3.4.2 h3422bc3_5 conda-forge libgettextpo 0.22.5 h8fbad5d_2 conda-forge libgettextpo-devel 0.22.5 h8fbad5d_2 conda-forge libgfortran 5.0.0 12_2_0_hd922786_32 conda-forge libgfortran5 12.2.0 h0eea778_32 conda-forge libiconv 1.17 h0d3ecfb_2 conda-forge libintl 0.22.5 h8fbad5d_2 conda-forge libintl-devel 0.22.5 h8fbad5d_2 conda-forge libjpeg-turbo 3.0.0 hb547adb_1 conda-forge liblapack 3.9.0 17_osxarm64_openblas conda-forge liblief 0.14.1 h3f3aa29_1 conda-forge libllvm14 14.0.6 hd1a9a77_4 conda-forge libllvm15 15.0.7 h2621b3d_4 conda-forge libllvm16 16.0.6 haab561b_3 conda-forge libllvm18 18.1.3 h30cc82d_0 conda-forge libmamba 1.5.8 h90c426b_0 conda-forge libmambapy 1.5.8 py310h5e0a2f6_0 conda-forge libnghttp2 1.58.0 ha4dd798_1 conda-forge libopenblas 0.3.23 openmp_hc731615_0 conda-forge libpng 1.6.43 h091b4b1_0 conda-forge libsodium 1.0.18 h27ca646_1 conda-forge libsolv 0.7.28 h1059232_2 conda-forge libsqlite 3.45.2 h091b4b1_0 conda-forge libssh2 1.11.0 h7a5bd25_0 conda-forge libtiff 4.6.0 h07db509_3 conda-forge libwebp-base 1.3.2 h93a5062_1 conda-forge libxcb 1.15 hf346824_0 conda-forge libxml2 2.12.6 h0d0cfa8_1 conda-forge libzlib 1.2.13 h53f4e23_5 conda-forge license-expression 30.1.1 pyhd8ed1ab_0 conda-forge llvm-openmp 18.1.3 hcd81f8e_0 conda-forge lz4-c 1.9.4 hb7217d7_0 conda-forge lzo 2.10 h642e427_1000 conda-forge mamba 1.5.8 py310ha5d4528_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.5 py310hd125d64_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge menuinst 2.0.2 py310hbe9552e_0 conda-forge more-itertools 10.2.0 pyhd8ed1ab_0 conda-forge msrest 0.7.1 pyhd8ed1ab_0 conda-forge nbformat 5.10.4 pyhd8ed1ab_0 conda-forge ncurses 6.4.20240210 h078ce10_0 conda-forge numpy 1.26.4 py310hd45542a_0 conda-forge oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge openjpeg 2.5.2 h9f1df11_0 conda-forge openssl 3.2.1 h0d3ecfb_1 conda-forge packaging 24.0 pyhd8ed1ab_0 conda-forge parso 0.8.4 pyhd8ed1ab_0 conda-forge patch 2.7.6 h27ca646_1002 conda-forge pathspec 0.12.1 pyhd8ed1ab_0 conda-forge pcre2 10.43 h26f9a81_0 conda-forge perl 5.32.1 7_h4614cfb_perl5 conda-forge pexpect 4.9.0 pyhd8ed1ab_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.3.0 py310h81a8c2e_0 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pkginfo 1.10.0 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge platformdirs 4.2.0 pyhd8ed1ab_0 conda-forge pluggy 1.4.0 pyhd8ed1ab_0 conda-forge progressbar2 4.4.2 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.42 pyha770c72_0 conda-forge prompt_toolkit 3.0.42 hd8ed1ab_0 conda-forge psutil 5.9.8 py310hd125d64_0 conda-forge pthread-stubs 0.4 h27ca646_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge py-lief 0.14.1 py310h1c30a33_1 conda-forge pybind11-abi 4 hd8ed1ab_3 conda-forge pycosat 0.6.6 py310h2aa6e3c_0 conda-forge pycparser 2.22 pyhd8ed1ab_0 conda-forge pycryptodome 3.20.0 py310hb53ef7b_0 conda-forge pydantic 2.6.4 pyhd8ed1ab_0 conda-forge pydantic-core 2.16.3 py310hd442715_0 conda-forge pygithub 2.3.0 pyhd8ed1ab_0 conda-forge pygments 2.17.2 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_1 conda-forge pynacl 1.5.0 py310h2aa6e3c_3 conda-forge pyopenssl 24.0.0 pyhd8ed1ab_0 conda-forge pyperclip 1.8.2 pyhd8ed1ab_2 conda-forge pyproject_hooks 1.0.0 pyhd8ed1ab_0 conda-forge pyrsistent 0.20.0 py310hd125d64_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.10.13 h2469fbe_1_cpython conda-forge python-build 1.2.1 pyhd8ed1ab_0 conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.19.1 pyhd8ed1ab_0 conda-forge python-libarchive-c 5.1 py310hbe9552e_0 conda-forge python-utils 3.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.10 4_cp310 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pyyaml 6.0.1 py310h2aa6e3c_1 conda-forge rapidfuzz 3.8.1 py310h692a8b6_0 conda-forge readline 8.2 h92ec313_1 conda-forge referencing 0.34.0 pyhd8ed1ab_0 conda-forge reproc 14.2.4.post0 h93a5062_1 conda-forge reproc-cpp 14.2.4.post0 h965bd2d_1 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 2.0.0 pyhd8ed1ab_0 conda-forge requests-toolbelt 1.0.0 pyhd8ed1ab_0 conda-forge rich 13.7.1 pyhd8ed1ab_0 conda-forge ripgrep 14.1.0 h5ef7bb8_0 conda-forge rpds-py 0.18.0 py310hf632f72_0 conda-forge ruamel.yaml 0.18.6 py310hd125d64_0 conda-forge ruamel.yaml.clib 0.2.8 py310hd125d64_0 conda-forge ruamel.yaml.jinja2 0.2.4 py_1 conda-forge ruamel_yaml 0.15.80 py310h2aa6e3c_1009 conda-forge scrypt 0.8.24 py310h76a31a2_0 conda-forge setuptools 69.2.0 pyhd8ed1ab_0 conda-forge shellingham 1.5.4 pyhd8ed1ab_0 conda-forge shyaml 0.6.2 pyhd3deb0d_0 conda-forge sigtool 0.1.3 h44b9a77_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge smmap 5.0.0 pyhd8ed1ab_0 conda-forge sniffio 1.3.1 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge stdlib-list 0.10.0 pyhd8ed1ab_0 conda-forge tapi 1100.0.11 he4954df_0 conda-forge tk 8.6.13 h5083fa2_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tomli-w 1.0.0 pyhd8ed1ab_0 conda-forge tomlkit 0.12.4 pyha770c72_0 conda-forge toolz 0.12.1 pyhd8ed1ab_0 conda-forge tornado 6.4 py310hd125d64_0 conda-forge tqdm 4.66.2 pyhd8ed1ab_0 conda-forge traitlets 5.14.2 pyhd8ed1ab_0 conda-forge trove-classifiers 2024.3.25 pyhd8ed1ab_0 conda-forge truststore 0.8.0 pyhd8ed1ab_0 conda-forge typer 0.12.2 pyhd8ed1ab_0 conda-forge typer-slim 0.12.2 pyhd8ed1ab_0 conda-forge typer-slim-standard 0.12.2 hd8ed1ab_0 conda-forge typing-extensions 4.11.0 hd8ed1ab_0 conda-forge typing_extensions 4.11.0 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge urllib3 2.2.1 pyhd8ed1ab_0 conda-forge userpath 1.7.0 pyhd8ed1ab_0 conda-forge virtualenv 20.25.1 pyhd8ed1ab_0 conda-forge vsts-python-api 0.1.22 py_0 conda-forge watchgod 0.8.2 pyhd8ed1ab_0 conda-forge wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge wrapt 1.16.0 py310hd125d64_0 conda-forge xorg-libxau 1.0.11 hb547adb_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xz 5.2.6 h57fd34a_0 conda-forge yaml 0.2.5 h3422bc3_2 conda-forge yaml-cpp 0.8.0 h13dd4ca_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zstandard 0.22.0 py310h6289e41_0 conda-forge zstd 1.5.5 h4f39d0f_0 conda-forge ```

Command result:

CONDA_DEBUG=1 conda search 'conda-forge::cuda-tools[subdir=win-64]=12.4.1'

```zsh % CONDA_DEBUG=1 conda search 'conda-forge::cuda-tools[subdir=win-64]=12.4.1' DEBUG conda.gateways.logging:set_log_level(233): log_level set to 10 Loading channels: ...working... DEBUG conda.core.package_cache_data:_check_writable(321): package cache directory '/Users/jkirkham/miniforge/pkgs' writable: True DEBUG conda.gateways.repodata:fetch_latest(827): Using cached repodata for https://conda.anaconda.org/conda-forge/win-64/repodata.json at /Users/jkirkham/miniforge/pkgs/cache/5afe41e9.json. Timeout in 1899 sec DEBUG conda.gateways.repodata:read_cache(907): Loading raw json for https://conda.anaconda.org/conda-forge/win-64/repodata.json at /Users/jkirkham/miniforge/pkgs/cache/5afe41e9.json done No match found for: conda-forge/win-64::cuda-tools=12.4.1. Search: conda-forge/win-64::*cuda-tools*=12.4.1 Traceback (most recent call last): File "/Users/jkirkham/miniforge/lib/python3.10/site-packages/conda/exception_handler.py", line 17, in __call__ return func(*args, **kwargs) File "/Users/jkirkham/miniforge/lib/python3.10/site-packages/conda/cli/main.py", line 83, in main_subshell exit_code = do_call(args, parser) File "/Users/jkirkham/miniforge/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 199, in do_call result = getattr(module, func_name)(args, parser) File "/Users/jkirkham/miniforge/lib/python3.10/site-packages/conda/cli/main_search.py", line 265, in execute raise PackagesNotFoundError((str(spec),), channels_urls) conda.exceptions.PackagesNotFoundError: The following packages are not available from current channels: - conda-forge/win-64::cuda-tools=12.4.1 Current channels: - https://conda.anaconda.org/conda-forge/win-64 - https://conda.anaconda.org/conda-forge/noarch To search for alternate channels that may provide the conda package you're looking for, navigate to https://anaconda.org and use the search bar at the top of the page. ```

dholth commented 3 months ago

Let's try reducing the cache TTL. CONDA_LOCAL_REPODATA_TTL=0 CONDA_DEBUG=1 conda search 'conda-forge::cuda-tools[subdir=win-64]=12.4.1' --json

jakirkham commented 3 months ago

That looks promising

CONDA_LOCAL_REPODATA_TTL=0 CONDA_DEBUG=1 conda search 'conda-forge::cuda-tools[subdir=win-64]=12.4.1' --json

``` DEBUG conda.gateways.logging:set_log_level(233): log_level set to 10 DEBUG conda.core.package_cache_data:_check_writable(321): package cache directory '/Users/jkirkham/miniforge/pkgs' writable: True DEBUG conda.gateways.repodata:fetch_latest(836): Local cache timed out for https://conda.anaconda.org/conda-forge/win-64/repodata.json at /Users/jkirkham/miniforge/pkgs/cache/5afe41e9.json DEBUG conda.gateways.repodata.jlap.interface:__init__(41): Using ZstdRepoInterface DEBUG conda.gateways.connection.session:add_binstar_token(247): Adding anaconda token for url DEBUG urllib3.connectionpool:_new_conn(1055): Starting new HTTPS connection (1): conda.anaconda.org:443 DEBUG urllib3.connectionpool:_make_request(549): https://conda.anaconda.org:443 "GET /t/ja-0b7702b5-09c2-4eba-86f1-afde7a4f059b/conda-forge/win-64/repodata.json.zst HTTP/1.1" 200 21595511 DEBUG conda.gateways.repodata.jlap.fetch:download_and_hash(243): https://conda.anaconda.org/conda-forge/win-64/repodata.json.zst {'Date': 'Wed, 10 Apr 2024 00:06:12 GMT', 'Content-Type': 'binary/octet-stream', 'Content-Length': '21595511', 'Connection': 'keep-alive', 'CF-Ray': '871e7d54f85f6b05-DFW', 'CF-Cache-Status': 'HIT', 'Accept-Ranges': 'bytes', 'Age': '624', 'Cache-Control': 'public, max-age=1200', 'ETag': '"265bb4f741e3a511d1b7fd13e0edee29"', 'Expires': 'Wed, 10 Apr 2024 00:26:12 GMT', 'Last-Modified': 'Tue, 09 Apr 2024 23:51:43 GMT', 'Vary': 'Accept-Encoding', 'x-amz-id-2': 'KsKq6Hrja0c+NuJOtbl7K2aZzL0wG6RjgNYYmQdqWpd3z3J4QGMToZG3p6uCst6RCFsGipoI7hk=', 'x-amz-request-id': '0TEK1J904ZF6JPAZ', 'x-amz-version-id': 'null', 'Set-Cookie': '__cf_bm=wiFBhoz93urWirCFc2ZG6eza4dPUj0Boq5Om5XXAAqg-1712707572-1.0.1.1-fImsderM0uAGUXTU1v5f2D6oevTo0EiN9RzvfapVIAJg8tz5A7xMZcag9R2Dk6LX_2rNKVn8hm_yraEI3GsI6RuELD2gizrmaUR0prPSIoM; path=/; expires=Wed, 10-Apr-24 00:36:12 GMT; domain=.anaconda.org; HttpOnly; Secure; SameSite=None', 'Server': 'cloudflare'} INFO conda.gateways.repodata.jlap.fetch:download_and_hash(264): Download 21595511 bytes {'User-Agent': 'conda/24.3.0 requests/2.31.0 CPython/3.10.13 Darwin/23.4.0 OSX/14.4.1 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8', 'Accept-Encoding': 'gzip, deflate, br, zstd', 'Accept': '*/*', 'Connection': 'keep-alive', 'if-none-match': '"6ca41bfb39370baa2abf1c3307d43f6e"'} DEBUG conda.gateways.repodata.jlap.fetch:timeme(196): Download complete https://conda.anaconda.org/conda-forge/win-64/repodata.json Took 1.14s INFO conda.gateways.repodata.jlap.fetch:request_url_jlap_state(431): Apply 0 patches 3796a3c1f32f8782… → 3796a3c1f32f8782… DEBUG conda.gateways.repodata.jlap.fetch:timeme(196): Apply Patches Took 0.00s ``` ```json { "cuda-tools": [ { "arch": null, "build": "h57928b3_0", "build_number": 0, "channel": "https://conda.anaconda.org/conda-forge/win-64", "constrains": [], "depends": [ "cuda-command-line-tools 12.4.1.*", "cuda-visual-tools 12.4.1.*" ], "fn": "cuda-tools-12.4.1-h57928b3_0.conda", "license": "LicenseRef-NVIDIA-End-User-License-Agreement", "md5": "12168c96051bbbc85dbb96ec9a8b0fac", "name": "cuda-tools", "platform": null, "sha256": "720c78b89fe36e19881347d69329f0ff8c1c4cd1630d39f3147d4fe08ace8583", "size": 20367, "subdir": "win-64", "timestamp": 1712694515857, "url": "https://conda.anaconda.org/conda-forge/win-64/cuda-tools-12.4.1-h57928b3_0.conda", "version": "12.4.1" } ] } ```

jakirkham commented 3 months ago

This case is approaching the 2hr mark, but does not appear to be picked up by CDN

Can find 1 of the 2 packages expected with CONDA_LOCAL_REPODATA_TTL=0 CONDA_DEBUG=1 conda search 'conda-forge::cuda-toolkit=12.4.1' --json

chenghlee commented 3 months ago

It looks like both packages are now available:

$ CONDA_LOCAL_REPODATA_TTL=0 CONDA_DEBUG=1 conda search 'conda-forge::cuda-toolkit=12.4.1' --json
DEBUG conda.gateways.logging:set_log_level(233): log_level set to 10
DEBUG conda.core.package_cache_data:_check_writable(321): package cache directory '/home/chl/.miniconda3-x86_64/pkgs' writable: True
DEBUG conda.gateways.repodata:fetch_latest(836): Local cache timed out for https://conda.anaconda.org/conda-forge/linux-64/repodata.json at /home/chl/.miniconda3-x86_64/pkgs/cache/497deca9.json
DEBUG conda.gateways.repodata.jlap.interface:__init__(41): Using ZstdRepoInterface
DEBUG conda.gateways.connection.session:add_binstar_token(247): Adding anaconda token for url <https://conda.anaconda.org/conda-forge/linux-64/repodata.json.zst>
DEBUG urllib3.connectionpool:_new_conn(1052): Starting new HTTPS connection (1): conda.anaconda.org:443
DEBUG urllib3.connectionpool:_make_request(546): https://conda.anaconda.org:443 "GET /t/ch-aad2628c-7f2b-496e-98ce-4fcff0ee47b9/conda-forge/linux-64/repodata.json.zst HTTP/1.1" 304 0
DEBUG conda.gateways.repodata.jlap.fetch:download_and_hash(243): https://conda.anaconda.org/conda-forge/linux-64/repodata.json.zst {'Date': 'Wed, 10 Apr 2024 17:33:49 GMT', 'Connection': 'keep-alive', 'CF-Ray': '87247bf1db4f6c79-DFW', 'CF-Cache-Status': 'HIT', 'Age': '1269', 'Cache-Control': 'public, max-age=1200', 'ETag': '"6134ed6707255ece9635559af12210f1"', 'Expires': 'Wed, 10 Apr 2024 17:53:49 GMT', 'Last-Modified': 'Wed, 10 Apr 2024 17:12:01 GMT', 'Vary': 'Accept-Encoding', 'x-amz-id-2': 'xXMe6GDWbJycN4+8p/BlbLkrY6hf+S8S8Ej7yaJjhjhoMx0RUMMY1OaU07Gb/yxsghhefqFxJqY=', 'x-amz-request-id': '4FRF0VNSKWS1SETG', 'x-amz-version-id': 'null', 'Set-Cookie': '__cf_bm=Zd4sY9TOVRGf_xkAFGWxZt22Jf9zV1CW1ZFappB_tHA-1712770429-1.0.1.1-Whna4UmYB02PdyGWm2hAjfjtN423iI9pahjBjPvhd19Y7Xir_WzKBEVZcHl7rVbWY94oHRFp5TOQeXEby1S1WrcfKFyABRrPsIO9Fo390pE; path=/; expires=Wed, 10-Apr-24 18:03:49 GMT; domain=.anaconda.org; HttpOnly; Secure; SameSite=None', 'Server': 'cloudflare'}
INFO conda.gateways.repodata.jlap.fetch:download_and_hash(264): Download 0 bytes {'User-Agent': 'conda/24.3.0 requests/2.31.0 CPython/3.11.7 Linux/6.5.0-26-generic linuxmint/21.3 glibc/2.35 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8 aau/0.4.4', 'Accept-Encoding': 'gzip, deflate, br, zstd', 'Accept': '*/*', 'Connection': 'keep-alive', 'if-none-match': '"6134ed6707255ece9635559af12210f1"'}
DEBUG conda.gateways.repodata.jlap.fetch:timeme(196): Download complete https://conda.anaconda.org/conda-forge/linux-64/repodata.json Took 0.09s
INFO conda.gateways.repodata.jlap.fetch:request_url_jlap_state(431): Apply 0 patches 4e79e31048896620… → 4e79e31048896620…
DEBUG conda.gateways.repodata.jlap.fetch:timeme(196): Apply Patches Took 0.00s
DEBUG conda.gateways.repodata:fetch_latest(836): Local cache timed out for https://conda.anaconda.org/conda-forge/noarch/repodata.json at /home/chl/.miniconda3-x86_64/pkgs/cache/09cdf8bf.json
DEBUG conda.gateways.repodata.jlap.interface:__init__(41): Using ZstdRepoInterface
DEBUG conda.gateways.connection.session:add_binstar_token(247): Adding anaconda token for url <https://conda.anaconda.org/conda-forge/noarch/repodata.json.zst>
DEBUG urllib3.connectionpool:_new_conn(1052): Starting new HTTPS connection (2): conda.anaconda.org:443
DEBUG urllib3.connectionpool:_make_request(546): https://conda.anaconda.org:443 "GET /t/ch-aad2628c-7f2b-496e-98ce-4fcff0ee47b9/conda-forge/noarch/repodata.json.zst HTTP/1.1" 304 0
DEBUG conda.gateways.repodata.jlap.fetch:download_and_hash(243): https://conda.anaconda.org/conda-forge/noarch/repodata.json.zst {'Date': 'Wed, 10 Apr 2024 17:33:52 GMT', 'Connection': 'keep-alive', 'CF-Ray': '87247c051e116b37-DFW', 'CF-Cache-Status': 'HIT', 'Age': '1272', 'Cache-Control': 'public, max-age=1200', 'ETag': '"3605ffb16e98f38bc071ce70257c459c"', 'Expires': 'Wed, 10 Apr 2024 17:53:52 GMT', 'Last-Modified': 'Wed, 10 Apr 2024 17:12:16 GMT', 'Vary': 'Accept-Encoding', 'x-amz-id-2': 'o7PiCRiTKh0i28OBv9US4lJb8Bne90hS+dQWjje44eZ8eGKGXf3KijXyyGIqkITCOeFXcv6WIwU=', 'x-amz-request-id': '4FR2PM0C9P3JSPP4', 'x-amz-version-id': 'null', 'Server': 'cloudflare'}
INFO conda.gateways.repodata.jlap.fetch:download_and_hash(264): Download 0 bytes {'User-Agent': 'conda/24.3.0 requests/2.31.0 CPython/3.11.7 Linux/6.5.0-26-generic linuxmint/21.3 glibc/2.35 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8 aau/0.4.4', 'Accept-Encoding': 'gzip, deflate, br, zstd', 'Accept': '*/*', 'Connection': 'keep-alive', 'if-none-match': '"3605ffb16e98f38bc071ce70257c459c"', 'Cookie': '__cf_bm=Zd4sY9TOVRGf_xkAFGWxZt22Jf9zV1CW1ZFappB_tHA-1712770429-1.0.1.1-Whna4UmYB02PdyGWm2hAjfjtN423iI9pahjBjPvhd19Y7Xir_WzKBEVZcHl7rVbWY94oHRFp5TOQeXEby1S1WrcfKFyABRrPsIO9Fo390pE'}
DEBUG conda.gateways.repodata.jlap.fetch:timeme(196): Download complete https://conda.anaconda.org/conda-forge/noarch/repodata.json Took 0.10s
INFO conda.gateways.repodata.jlap.fetch:request_url_jlap_state(431): Apply 0 patches 3a8d1a3f16e02a99… → 3a8d1a3f16e02a99…
DEBUG conda.gateways.repodata.jlap.fetch:timeme(196): Apply Patches Took 0.00s
{
  "cuda-toolkit": [
    {
      "arch": null,
      "build": "h7428d3b_0",
      "build_number": 0,
      "channel": "https://conda.anaconda.org/conda-forge/noarch",
      "constrains": [],
      "depends": [
        "__win",
        "cuda-compiler 12.4.1.*",
        "cuda-libraries 12.4.1.*",
        "cuda-libraries-dev 12.4.1.*",
        "cuda-nvml-dev 12.4.127.*",
        "cuda-tools 12.4.1.*"
      ],
      "fn": "cuda-toolkit-12.4.1-h7428d3b_0.conda",
      "license": "LicenseRef-NVIDIA-End-User-License-Agreement",
      "md5": "da683653aaadbeeb5a7a2561b2464728",
      "name": "cuda-toolkit",
      "noarch": "generic",
      "package_type": "noarch_generic",
      "platform": null,
      "sha256": "3f750755a089f61fb58e6e255706bd2d010fd217fcb0a503c2de9fe3a337b247",
      "size": 20519,
      "subdir": "noarch",
      "timestamp": 1712711684371,
      "url": "https://conda.anaconda.org/conda-forge/noarch/cuda-toolkit-12.4.1-h7428d3b_0.conda",
      "version": "12.4.1"
    },
    {
      "arch": null,
      "build": "ha804496_0",
      "build_number": 0,
      "channel": "https://conda.anaconda.org/conda-forge/noarch",
      "constrains": [],
      "depends": [
        "__linux",
        "cuda-compiler 12.4.1.*",
        "cuda-libraries 12.4.1.*",
        "cuda-libraries-dev 12.4.1.*",
        "cuda-nvml-dev 12.4.127.*",
        "cuda-tools 12.4.1.*"
      ],
      "fn": "cuda-toolkit-12.4.1-ha804496_0.conda",
      "license": "LicenseRef-NVIDIA-End-User-License-Agreement",
      "md5": "e1e8cfdbb172f4b6558ce2db688e851f",
      "name": "cuda-toolkit",
      "noarch": "generic",
      "package_type": "noarch_generic",
      "platform": null,
      "sha256": "c3faecbb52cbdb82d1723e82ed572ae23bc187863b7f499f8afe7247cf1178c1",
      "size": 20097,
      "subdir": "noarch",
      "timestamp": 1712717191443,
      "url": "https://conda.anaconda.org/conda-forge/noarch/cuda-toolkit-12.4.1-ha804496_0.conda",
      "version": "12.4.1"
    }
  ]
}

This latest delay is most likely due to the issues with the anaconda.org backend (xref: conda/infrastructure#899). The channel cloning process relies on various calls to .org's API; the .org database was sporadically triggering the OOM (out-of-memory) killer on the backend host. Anaconda's infrastructure team has expanded memory and scaling allocation for the database backend, and that should help stabilize things again.

dholth commented 3 months ago

Could we write a process that periodically checks repodata-clone.json versus repodata.json to keep track of any delay between packages appearing in the former versus the latter

jakirkham commented 3 months ago

Thanks Cheng and Daniel! 🙏

Daniel, think that is a good idea. If there is some way to share log details or maybe graphs on resource usage, that might help as well

We were also wondering if it would make sense to have a GH template for CDN issues ( https://github.com/conda/infrastructure/issues/912 ). Are there specific pieces of info we should be capturing that would help narrow things down?

astrofrog commented 2 months ago

Seeing issues today with https://anaconda.org/conda-forge/hyperion-fortran/files, not available after 2h

jakirkham commented 1 month ago

Am seeing this with recent llvmlite RC packages. These were upload ~2hrs ago. However it looks like none are available yet

jjerphan commented 1 month ago

To know whether recent packages can be pulled, one can watch the last update of repodata.json with:

watch -n 1 "curl -sI https://conda.anaconda.org/conda-forge/linux-64/repodata.json | grep 'last-modified'"

conda / infrastructure

Slow CDN mirroring #892