search

conda / infrastructure

A repo to report issues and have discussions about the conda infrastructure
BSD 3-Clause "New" or "Revised" License
11 stars 15 forks source link

`.conda` packages uploaded to `nvidia` channel not mirroring to CDN #945

Open jakirkham opened 2 months ago

jakirkham commented 2 months ago

Recently we uploaded .conda packages for CUDA 12.5 to the nvidia channel. However we discovered they were not mirroring to CDN.

We noticed this by trying to install CUDA 12.5 from the nvidia channel, but got the following error:

$ conda create -c nvidia -n cuda12.5 cuda=12.5
Channels:
 - nvidia
 - defaults
Platform: linux-aarch64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - cuda=12.5*

Current channels:

  - https://conda.anaconda.org/nvidia
  - defaults

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

However I can see the package is available via conda search

$ conda search 'nvidia/label/main::cuda[subdir=linux-aarch64]=12.5' 
Loading channels: done
# Name                       Version           Build  Channel             
cuda                          12.5.0               0  nvidia/label/main

Also via Anaconda.org

Screenshot 2024-05-22 at 1 54 58 PM

Since these were uploaded, we were recommended to upload .tar.bz2 packages to the nvidia channel, which the CDN appears to handle better

Still it would be good to support .conda packages on the nvidia channel CDN too. This issue remains open to address that

jakirkham commented 2 months ago

An interesting wrinkle is dropping the /label/main bit or use the -c channel syntax with conda search, it returns no packages found. For example

$ conda search --skip-flexible-search -c nvidia 'cuda[subdir=linux-aarch64]=12.5' 
Loading channels: done

PackagesNotFoundError: The following packages are not available from current channels:

  - cuda=12.5[subdir=linux-aarch64]

Current channels:

  - https://conda.anaconda.org/nvidia/linux-aarch64
  - https://conda.anaconda.org/nvidia/noarch
  - https://repo.anaconda.com/pkgs/main/linux-aarch64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-aarch64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Note: Flexible search is skipped otherwise we pick up cuda-python, which is not what we are looking for here

jakirkham commented 2 months ago

cc @adibbley (for vis)

jaimergp commented 2 months ago

Maybe adding label/main bypasses the CDN, where it is missing?

jakirkham commented 2 months ago

That is what I'm thinking as well

For example, the same conda create command above using nvidia/label/main works

``` conda create -c nvidia/label/main -n cuda12.5 cuda=12.5 --dry-run Channels: - nvidia/label/main - defaults Platform: linux-aarch64 Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: /opt/conda/envs/cuda12.5 added / updated specs: - cuda=12.5 The following packages will be downloaded: package | build ---------------------------|----------------- _libgcc_mutex-0.1 | main 2 KB defaults _openmp_mutex-5.1 | 51_gnu 1.4 MB defaults binutils_impl_linux-aarch64-2.38| h0c9fd12_1 5.5 MB defaults binutils_linux-aarch64-2.38.0| h0c7b1b4_0 24 KB defaults bzip2-1.0.8 | h998d150_6 210 KB defaults ca-certificates-2024.3.11 | hd43f75c_0 127 KB defaults cuda-12.5.0 | 0 17 KB nvidia/label/main cuda-cccl-12.5.39 | 0 16 KB nvidia/label/main cuda-cccl_linux-aarch64-12.5.39| 0 1.3 MB nvidia/label/main cuda-command-line-tools-12.5.0| 0 17 KB nvidia/label/main cuda-compiler-12.5.0 | 0 17 KB nvidia/label/main cuda-crt-dev_linux-aarch64-12.5.40| 0 79 KB nvidia/label/main cuda-crt-tools-12.5.40 | 0 19 KB nvidia/label/main cuda-cudart-12.5.39 | 0 17 KB nvidia/label/main cuda-cudart-dev-12.5.39 | 0 17 KB nvidia/label/main cuda-cudart-dev_linux-aarch64-12.5.39| 0 352 KB nvidia/label/main cuda-cudart-static-12.5.39 | 0 17 KB nvidia/label/main cuda-cudart-static_linux-aarch64-12.5.39| 0 715 KB nvidia/label/main cuda-cudart_linux-aarch64-12.5.39| 0 191 KB nvidia/label/main cuda-cuobjdump-12.5.39 | 0 242 KB nvidia/label/main cuda-cupti-12.5.39 | 0 1.6 MB nvidia/label/main cuda-cupti-dev-12.5.39 | 0 6.4 MB nvidia/label/main cuda-cuxxfilt-12.5.39 | 0 202 KB nvidia/label/main cuda-driver-dev-12.5.39 | 0 17 KB nvidia/label/main cuda-driver-dev_linux-aarch64-12.5.39| 0 32 KB nvidia/label/main cuda-gdb-12.5.39 | 0 5.1 MB nvidia/label/main cuda-libraries-12.5.0 | 0 17 KB nvidia/label/main cuda-libraries-dev-12.5.0 | 0 17 KB nvidia/label/main cuda-nvcc-12.5.40 | 0 17 KB nvidia/label/main cuda-nvcc-dev_linux-aarch64-12.5.40| 0 10.8 MB nvidia/label/main cuda-nvcc-impl-12.5.40 | 0 18 KB nvidia/label/main cuda-nvcc-tools-12.5.40 | 0 19.2 MB nvidia/label/main cuda-nvcc_linux-aarch64-12.5.40| 0 18 KB nvidia/label/main cuda-nvdisasm-12.5.39 | 0 47.6 MB nvidia/label/main cuda-nvml-dev-12.5.39 | 0 158 KB nvidia/label/main cuda-nvprune-12.5.39 | 0 56 KB nvidia/label/main cuda-nvrtc-12.5.40 | 0 17.8 MB nvidia/label/main cuda-nvrtc-dev-12.5.40 | 0 27 KB nvidia/label/main cuda-nvtx-12.5.39 | 0 28 KB nvidia/label/main cuda-nvvm-dev_linux-aarch64-12.5.40| 0 17 KB nvidia/label/main cuda-nvvm-impl-12.5.40 | 0 8.4 MB nvidia/label/main cuda-nvvm-tools-12.5.40 | 0 10.8 MB nvidia/label/main cuda-profiler-api-12.5.39 | 0 19 KB nvidia/label/main cuda-runtime-12.5.0 | 0 17 KB nvidia/label/main cuda-sanitizer-api-12.5.39 | 0 9.8 MB nvidia/label/main cuda-toolkit-12.5.0 | 0 17 KB nvidia/label/main cuda-tools-12.5.0 | 0 17 KB nvidia/label/main cuda-version-12.5 | 3 17 KB nvidia/label/main cuda-visual-tools-12.5.0 | 0 17 KB nvidia/label/main dbus-1.13.18 | h821dc26_0 544 KB defaults expat-2.6.2 | h419075a_0 159 KB defaults fontconfig-2.14.1 | h652894d_2 323 KB defaults freetype-2.12.1 | h6df46f4_0 632 KB defaults gcc_impl_linux-aarch64-11.2.0| h1234567_1 21.0 MB defaults gcc_linux-aarch64-11.2.0 | h3fc98b3_0 25 KB defaults gds-tools-1.10.0.4 | 0 37.8 MB nvidia/label/main glib-2.78.4 | h419075a_0 517 KB defaults glib-tools-2.78.4 | h419075a_0 126 KB defaults gmp-6.2.1 | h22f4aa5_3 515 KB defaults gxx_impl_linux-aarch64-11.2.0| h1234567_1 10.2 MB defaults gxx_linux-aarch64-11.2.0 | h0c7b1b4_0 24 KB defaults icu-73.1 | h419075a_0 26.2 MB defaults kernel-headers_linux-aarch64-4.14.214| h1cb168f_14 1018 KB defaults ld_impl_linux-aarch64-2.38 | h8131f2d_1 778 KB defaults libcublas-12.5.2.13 | 0 240.2 MB nvidia/label/main libcublas-dev-12.5.2.13 | 0 71 KB nvidia/label/main libcufft-11.2.3.18 | 0 148.6 MB nvidia/label/main libcufft-dev-11.2.3.18 | 0 28 KB nvidia/label/main libcufile-1.10.0.4 | 0 843 KB nvidia/label/main libcufile-dev-1.10.0.4 | 0 29 KB nvidia/label/main libcurand-10.3.6.39 | 0 39.7 MB nvidia/label/main libcurand-dev-10.3.6.39 | 0 245 KB nvidia/label/main libcusolver-11.6.2.40 | 0 79.0 MB nvidia/label/main libcusolver-dev-11.6.2.40 | 0 56 KB nvidia/label/main libcusparse-12.4.1.24 | 0 113.8 MB nvidia/label/main libcusparse-dev-12.4.1.24 | 0 41 KB nvidia/label/main libffi-3.4.4 | h419075a_1 140 KB defaults libgcc-devel_linux-aarch64-11.2.0| h1234567_1 311 KB defaults libgcc-ng-11.2.0 | h1234567_1 1.3 MB defaults libglib-2.78.4 | hd439bcf_0 1.6 MB defaults libgomp-11.2.0 | h1234567_1 466 KB defaults libiconv-1.16 | h998d150_3 772 KB defaults libnpp-12.3.0.116 | 0 94.5 MB nvidia/label/main libnpp-dev-12.3.0.116 | 0 460 KB nvidia/label/main libnvfatbin-12.5.39 | 0 737 KB nvidia/label/main libnvfatbin-dev-12.5.39 | 0 21 KB nvidia/label/main libnvjitlink-12.5.40 | 0 15.6 MB nvidia/label/main libnvjitlink-dev-12.5.40 | 0 20 KB nvidia/label/main libnvjpeg-12.3.2.38 | 0 2.3 MB nvidia/label/main libnvjpeg-dev-12.3.2.38 | 0 27 KB nvidia/label/main libpng-1.6.39 | h998d150_0 320 KB defaults libstdcxx-devel_linux-aarch64-11.2.0| h1234567_1 2.7 MB defaults libstdcxx-ng-11.2.0 | h1234567_1 779 KB defaults libuuid-1.41.5 | h998d150_0 29 KB defaults libxcb-1.15 | h2f4d8fa_0 484 KB defaults libxkbcommon-1.0.1 | h998d150_1 601 KB defaults libxml2-2.10.4 | h6097fa9_2 808 KB defaults ncurses-6.4 | h419075a_0 1.1 MB defaults nsight-compute-2024.2.0.16 | 2 227.6 MB nvidia/label/main nspr-4.35 | h419075a_0 251 KB defaults nss-3.89.1 | h419075a_0 2.1 MB defaults openssl-3.0.13 | h2f4d8fa_2 5.3 MB defaults pcre2-10.42 | hcfaa891_1 1.3 MB defaults pip-24.0 | py312hd43f75c_0 3.3 MB defaults python-3.12.3 | h8edadfe_1 14.2 MB defaults readline-8.2 | h998d150_0 381 KB defaults setuptools-69.5.1 | py312hd43f75c_0 1.3 MB defaults sqlite-3.45.3 | h998d150_0 1.5 MB defaults sysroot_linux-aarch64-2.26 | h4744778_14 11.3 MB defaults tk-8.6.14 | h987d8db_0 3.5 MB defaults tzdata-2024a | h04d1e81_0 116 KB defaults wheel-0.43.0 | py312hd43f75c_0 142 KB defaults xz-5.4.6 | h998d150_1 662 KB defaults zlib-1.2.13 | h998d150_1 113 KB defaults ------------------------------------------------------------ Total: 1.24 GB The following NEW packages will be INSTALLED: _libgcc_mutex pkgs/main/linux-aarch64::_libgcc_mutex-0.1-main _openmp_mutex pkgs/main/linux-aarch64::_openmp_mutex-5.1-51_gnu binutils_impl_lin~ pkgs/main/linux-aarch64::binutils_impl_linux-aarch64-2.38-h0c9fd12_1 binutils_linux-aa~ pkgs/main/linux-aarch64::binutils_linux-aarch64-2.38.0-h0c7b1b4_0 bzip2 pkgs/main/linux-aarch64::bzip2-1.0.8-h998d150_6 ca-certificates pkgs/main/linux-aarch64::ca-certificates-2024.3.11-hd43f75c_0 cuda nvidia/label/main/linux-aarch64::cuda-12.5.0-0 cuda-cccl nvidia/label/main/linux-aarch64::cuda-cccl-12.5.39-0 cuda-cccl_linux-a~ nvidia/label/main/noarch::cuda-cccl_linux-aarch64-12.5.39-0 cuda-command-line~ nvidia/label/main/linux-aarch64::cuda-command-line-tools-12.5.0-0 cuda-compiler nvidia/label/main/linux-aarch64::cuda-compiler-12.5.0-0 cuda-crt-dev_linu~ nvidia/label/main/noarch::cuda-crt-dev_linux-aarch64-12.5.40-0 cuda-crt-tools nvidia/label/main/linux-aarch64::cuda-crt-tools-12.5.40-0 cuda-cudart nvidia/label/main/linux-aarch64::cuda-cudart-12.5.39-0 cuda-cudart-dev nvidia/label/main/linux-aarch64::cuda-cudart-dev-12.5.39-0 cuda-cudart-dev_l~ nvidia/label/main/noarch::cuda-cudart-dev_linux-aarch64-12.5.39-0 cuda-cudart-static nvidia/label/main/linux-aarch64::cuda-cudart-static-12.5.39-0 cuda-cudart-stati~ nvidia/label/main/noarch::cuda-cudart-static_linux-aarch64-12.5.39-0 cuda-cudart_linux~ nvidia/label/main/noarch::cuda-cudart_linux-aarch64-12.5.39-0 cuda-cuobjdump nvidia/label/main/linux-aarch64::cuda-cuobjdump-12.5.39-0 cuda-cupti nvidia/label/main/linux-aarch64::cuda-cupti-12.5.39-0 cuda-cupti-dev nvidia/label/main/linux-aarch64::cuda-cupti-dev-12.5.39-0 cuda-cuxxfilt nvidia/label/main/linux-aarch64::cuda-cuxxfilt-12.5.39-0 cuda-driver-dev nvidia/label/main/linux-aarch64::cuda-driver-dev-12.5.39-0 cuda-driver-dev_l~ nvidia/label/main/noarch::cuda-driver-dev_linux-aarch64-12.5.39-0 cuda-gdb nvidia/label/main/linux-aarch64::cuda-gdb-12.5.39-0 cuda-libraries nvidia/label/main/linux-aarch64::cuda-libraries-12.5.0-0 cuda-libraries-dev nvidia/label/main/linux-aarch64::cuda-libraries-dev-12.5.0-0 cuda-nvcc nvidia/label/main/linux-aarch64::cuda-nvcc-12.5.40-0 cuda-nvcc-dev_lin~ nvidia/label/main/noarch::cuda-nvcc-dev_linux-aarch64-12.5.40-0 cuda-nvcc-impl nvidia/label/main/linux-aarch64::cuda-nvcc-impl-12.5.40-0 cuda-nvcc-tools nvidia/label/main/linux-aarch64::cuda-nvcc-tools-12.5.40-0 cuda-nvcc_linux-a~ nvidia/label/main/linux-aarch64::cuda-nvcc_linux-aarch64-12.5.40-0 cuda-nvdisasm nvidia/label/main/linux-aarch64::cuda-nvdisasm-12.5.39-0 cuda-nvml-dev nvidia/label/main/linux-aarch64::cuda-nvml-dev-12.5.39-0 cuda-nvprune nvidia/label/main/linux-aarch64::cuda-nvprune-12.5.39-0 cuda-nvrtc nvidia/label/main/linux-aarch64::cuda-nvrtc-12.5.40-0 cuda-nvrtc-dev nvidia/label/main/linux-aarch64::cuda-nvrtc-dev-12.5.40-0 cuda-nvtx nvidia/label/main/linux-aarch64::cuda-nvtx-12.5.39-0 cuda-nvvm-dev_lin~ nvidia/label/main/noarch::cuda-nvvm-dev_linux-aarch64-12.5.40-0 cuda-nvvm-impl nvidia/label/main/linux-aarch64::cuda-nvvm-impl-12.5.40-0 cuda-nvvm-tools nvidia/label/main/linux-aarch64::cuda-nvvm-tools-12.5.40-0 cuda-profiler-api nvidia/label/main/linux-aarch64::cuda-profiler-api-12.5.39-0 cuda-runtime nvidia/label/main/linux-aarch64::cuda-runtime-12.5.0-0 cuda-sanitizer-api nvidia/label/main/linux-aarch64::cuda-sanitizer-api-12.5.39-0 cuda-toolkit nvidia/label/main/linux-aarch64::cuda-toolkit-12.5.0-0 cuda-tools nvidia/label/main/linux-aarch64::cuda-tools-12.5.0-0 cuda-version nvidia/label/main/noarch::cuda-version-12.5-3 cuda-visual-tools nvidia/label/main/linux-aarch64::cuda-visual-tools-12.5.0-0 dbus pkgs/main/linux-aarch64::dbus-1.13.18-h821dc26_0 expat pkgs/main/linux-aarch64::expat-2.6.2-h419075a_0 fontconfig pkgs/main/linux-aarch64::fontconfig-2.14.1-h652894d_2 freetype pkgs/main/linux-aarch64::freetype-2.12.1-h6df46f4_0 gcc_impl_linux-aa~ pkgs/main/linux-aarch64::gcc_impl_linux-aarch64-11.2.0-h1234567_1 gcc_linux-aarch64 pkgs/main/linux-aarch64::gcc_linux-aarch64-11.2.0-h3fc98b3_0 gds-tools nvidia/label/main/linux-aarch64::gds-tools-1.10.0.4-0 glib pkgs/main/linux-aarch64::glib-2.78.4-h419075a_0 glib-tools pkgs/main/linux-aarch64::glib-tools-2.78.4-h419075a_0 gmp pkgs/main/linux-aarch64::gmp-6.2.1-h22f4aa5_3 gxx_impl_linux-aa~ pkgs/main/linux-aarch64::gxx_impl_linux-aarch64-11.2.0-h1234567_1 gxx_linux-aarch64 pkgs/main/linux-aarch64::gxx_linux-aarch64-11.2.0-h0c7b1b4_0 icu pkgs/main/linux-aarch64::icu-73.1-h419075a_0 kernel-headers_li~ pkgs/main/noarch::kernel-headers_linux-aarch64-4.14.214-h1cb168f_14 ld_impl_linux-aar~ pkgs/main/linux-aarch64::ld_impl_linux-aarch64-2.38-h8131f2d_1 libcublas nvidia/label/main/linux-aarch64::libcublas-12.5.2.13-0 libcublas-dev nvidia/label/main/linux-aarch64::libcublas-dev-12.5.2.13-0 libcufft nvidia/label/main/linux-aarch64::libcufft-11.2.3.18-0 libcufft-dev nvidia/label/main/linux-aarch64::libcufft-dev-11.2.3.18-0 libcufile nvidia/label/main/linux-aarch64::libcufile-1.10.0.4-0 libcufile-dev nvidia/label/main/linux-aarch64::libcufile-dev-1.10.0.4-0 libcurand nvidia/label/main/linux-aarch64::libcurand-10.3.6.39-0 libcurand-dev nvidia/label/main/linux-aarch64::libcurand-dev-10.3.6.39-0 libcusolver nvidia/label/main/linux-aarch64::libcusolver-11.6.2.40-0 libcusolver-dev nvidia/label/main/linux-aarch64::libcusolver-dev-11.6.2.40-0 libcusparse nvidia/label/main/linux-aarch64::libcusparse-12.4.1.24-0 libcusparse-dev nvidia/label/main/linux-aarch64::libcusparse-dev-12.4.1.24-0 libffi pkgs/main/linux-aarch64::libffi-3.4.4-h419075a_1 libgcc-devel_linu~ pkgs/main/linux-aarch64::libgcc-devel_linux-aarch64-11.2.0-h1234567_1 libgcc-ng pkgs/main/linux-aarch64::libgcc-ng-11.2.0-h1234567_1 libglib pkgs/main/linux-aarch64::libglib-2.78.4-hd439bcf_0 libgomp pkgs/main/linux-aarch64::libgomp-11.2.0-h1234567_1 libiconv pkgs/main/linux-aarch64::libiconv-1.16-h998d150_3 libnpp nvidia/label/main/linux-aarch64::libnpp-12.3.0.116-0 libnpp-dev nvidia/label/main/linux-aarch64::libnpp-dev-12.3.0.116-0 libnvfatbin nvidia/label/main/linux-aarch64::libnvfatbin-12.5.39-0 libnvfatbin-dev nvidia/label/main/linux-aarch64::libnvfatbin-dev-12.5.39-0 libnvjitlink nvidia/label/main/linux-aarch64::libnvjitlink-12.5.40-0 libnvjitlink-dev nvidia/label/main/linux-aarch64::libnvjitlink-dev-12.5.40-0 libnvjpeg nvidia/label/main/linux-aarch64::libnvjpeg-12.3.2.38-0 libnvjpeg-dev nvidia/label/main/linux-aarch64::libnvjpeg-dev-12.3.2.38-0 libpng pkgs/main/linux-aarch64::libpng-1.6.39-h998d150_0 libstdcxx-devel_l~ pkgs/main/linux-aarch64::libstdcxx-devel_linux-aarch64-11.2.0-h1234567_1 libstdcxx-ng pkgs/main/linux-aarch64::libstdcxx-ng-11.2.0-h1234567_1 libuuid pkgs/main/linux-aarch64::libuuid-1.41.5-h998d150_0 libxcb pkgs/main/linux-aarch64::libxcb-1.15-h2f4d8fa_0 libxkbcommon pkgs/main/linux-aarch64::libxkbcommon-1.0.1-h998d150_1 libxml2 pkgs/main/linux-aarch64::libxml2-2.10.4-h6097fa9_2 ncurses pkgs/main/linux-aarch64::ncurses-6.4-h419075a_0 nsight-compute nvidia/label/main/linux-aarch64::nsight-compute-2024.2.0.16-2 nspr pkgs/main/linux-aarch64::nspr-4.35-h419075a_0 nss pkgs/main/linux-aarch64::nss-3.89.1-h419075a_0 openssl pkgs/main/linux-aarch64::openssl-3.0.13-h2f4d8fa_2 pcre2 pkgs/main/linux-aarch64::pcre2-10.42-hcfaa891_1 pip pkgs/main/linux-aarch64::pip-24.0-py312hd43f75c_0 python pkgs/main/linux-aarch64::python-3.12.3-h8edadfe_1 readline pkgs/main/linux-aarch64::readline-8.2-h998d150_0 setuptools pkgs/main/linux-aarch64::setuptools-69.5.1-py312hd43f75c_0 sqlite pkgs/main/linux-aarch64::sqlite-3.45.3-h998d150_0 sysroot_linux-aar~ pkgs/main/noarch::sysroot_linux-aarch64-2.26-h4744778_14 tk pkgs/main/linux-aarch64::tk-8.6.14-h987d8db_0 tzdata pkgs/main/noarch::tzdata-2024a-h04d1e81_0 wheel pkgs/main/linux-aarch64::wheel-0.43.0-py312hd43f75c_0 xz pkgs/main/linux-aarch64::xz-5.4.6-h998d150_1 zlib pkgs/main/linux-aarch64::zlib-1.2.13-h998d150_1 DryRunExit: Dry run. Exiting. ```

So sounds like this is just an issue mirroring to CDN

jakirkham commented 2 months ago

cc @chenghlee (for vis)

chenghlee commented 2 months ago

Huh. Is still an issue on your end? It seems to be working for me, so if you're still seeing the issue, very likely some weird CDN thing.

``` $ conda create --dry-run --platform=linux-aarch64 --json -n __test__ \ --override-channels -c nvidia -c main cuda=12.5 | \ jq -rS '.actions.LINK|.[]|[.channel,.platform,.dist_name]|join("/")' | \ sort -t/ -k3 main/linux-aarch64/binutils_impl_linux-aarch64-2.38-h0c9fd12_1 main/linux-aarch64/binutils_linux-aarch64-2.38.0-h0c7b1b4_0 main/linux-aarch64/bzip2-1.0.8-h998d150_6 main/linux-aarch64/ca-certificates-2024.3.11-hd43f75c_0 nvidia/linux-aarch64/cuda-12.5.0-0 nvidia/linux-aarch64/cuda-cccl-12.5.39-0 nvidia/noarch/cuda-cccl_linux-aarch64-12.5.39-0 nvidia/linux-aarch64/cuda-command-line-tools-12.5.0-0 nvidia/linux-aarch64/cuda-compiler-12.5.0-0 nvidia/noarch/cuda-crt-dev_linux-aarch64-12.5.40-0 nvidia/linux-aarch64/cuda-crt-tools-12.5.40-0 nvidia/linux-aarch64/cuda-cudart-12.5.39-0 nvidia/linux-aarch64/cuda-cudart-dev-12.5.39-0 nvidia/noarch/cuda-cudart-dev_linux-aarch64-12.5.39-0 nvidia/noarch/cuda-cudart_linux-aarch64-12.5.39-0 nvidia/linux-aarch64/cuda-cudart-static-12.5.39-0 nvidia/noarch/cuda-cudart-static_linux-aarch64-12.5.39-0 nvidia/linux-aarch64/cuda-cuobjdump-12.5.39-0 nvidia/linux-aarch64/cuda-cupti-12.5.39-0 nvidia/linux-aarch64/cuda-cupti-dev-12.5.39-0 nvidia/linux-aarch64/cuda-cuxxfilt-12.5.39-0 nvidia/linux-aarch64/cuda-driver-dev-12.5.39-0 nvidia/noarch/cuda-driver-dev_linux-aarch64-12.5.39-0 nvidia/linux-aarch64/cuda-gdb-12.5.39-0 nvidia/linux-aarch64/cuda-libraries-12.5.0-0 nvidia/linux-aarch64/cuda-libraries-dev-12.5.0-0 nvidia/linux-aarch64/cuda-nvcc-12.5.40-0 nvidia/noarch/cuda-nvcc-dev_linux-aarch64-12.5.40-0 nvidia/linux-aarch64/cuda-nvcc-impl-12.5.40-0 nvidia/linux-aarch64/cuda-nvcc_linux-aarch64-12.5.40-0 nvidia/linux-aarch64/cuda-nvcc-tools-12.5.40-0 nvidia/linux-aarch64/cuda-nvdisasm-12.5.39-0 nvidia/linux-aarch64/cuda-nvml-dev-12.5.39-0 nvidia/linux-aarch64/cuda-nvprune-12.5.39-0 nvidia/linux-aarch64/cuda-nvrtc-12.5.40-0 nvidia/linux-aarch64/cuda-nvrtc-dev-12.5.40-0 nvidia/linux-aarch64/cuda-nvtx-12.5.39-0 nvidia/noarch/cuda-nvvm-dev_linux-aarch64-12.5.40-0 nvidia/linux-aarch64/cuda-nvvm-impl-12.5.40-0 nvidia/linux-aarch64/cuda-nvvm-tools-12.5.40-0 nvidia/linux-aarch64/cuda-profiler-api-12.5.39-0 nvidia/linux-aarch64/cuda-runtime-12.5.0-0 nvidia/linux-aarch64/cuda-sanitizer-api-12.5.39-0 nvidia/linux-aarch64/cuda-toolkit-12.5.0-0 nvidia/linux-aarch64/cuda-tools-12.5.0-0 nvidia/noarch/cuda-version-12.5-3 nvidia/linux-aarch64/cuda-visual-tools-12.5.0-0 main/linux-aarch64/dbus-1.13.18-h821dc26_0 main/linux-aarch64/expat-2.6.2-h419075a_0 main/linux-aarch64/fontconfig-2.14.1-h652894d_2 main/linux-aarch64/freetype-2.12.1-h6df46f4_0 main/linux-aarch64/gcc_impl_linux-aarch64-11.2.0-h1234567_1 main/linux-aarch64/gcc_linux-aarch64-11.2.0-h3fc98b3_0 nvidia/linux-aarch64/gds-tools-1.10.0.4-0 main/linux-aarch64/glib-2.78.4-h419075a_0 main/linux-aarch64/glib-tools-2.78.4-h419075a_0 main/linux-aarch64/gmp-6.2.1-h22f4aa5_3 main/linux-aarch64/gxx_impl_linux-aarch64-11.2.0-h1234567_1 main/linux-aarch64/gxx_linux-aarch64-11.2.0-h0c7b1b4_0 main/linux-aarch64/icu-73.1-h419075a_0 main/noarch/kernel-headers_linux-aarch64-4.14.214-h1cb168f_14 main/linux-aarch64/ld_impl_linux-aarch64-2.38-h8131f2d_1 nvidia/linux-aarch64/libcublas-12.5.2.13-0 nvidia/linux-aarch64/libcublas-dev-12.5.2.13-0 nvidia/linux-aarch64/libcufft-11.2.3.18-0 nvidia/linux-aarch64/libcufft-dev-11.2.3.18-0 nvidia/linux-aarch64/libcufile-1.10.0.4-0 nvidia/linux-aarch64/libcufile-dev-1.10.0.4-0 nvidia/linux-aarch64/libcurand-10.3.6.39-0 nvidia/linux-aarch64/libcurand-dev-10.3.6.39-0 nvidia/linux-aarch64/libcusolver-11.6.2.40-0 nvidia/linux-aarch64/libcusolver-dev-11.6.2.40-0 nvidia/linux-aarch64/libcusparse-12.4.1.24-0 nvidia/linux-aarch64/libcusparse-dev-12.4.1.24-0 main/linux-aarch64/libffi-3.4.4-h419075a_1 main/linux-aarch64/libgcc-devel_linux-aarch64-11.2.0-h1234567_1 main/linux-aarch64/_libgcc_mutex-0.1-main main/linux-aarch64/libgcc-ng-11.2.0-h1234567_1 main/linux-aarch64/libglib-2.78.4-hd439bcf_0 main/linux-aarch64/libgomp-11.2.0-h1234567_1 main/linux-aarch64/libiconv-1.16-h998d150_3 nvidia/linux-aarch64/libnpp-12.3.0.116-0 nvidia/linux-aarch64/libnpp-dev-12.3.0.116-0 nvidia/linux-aarch64/libnvfatbin-12.5.39-0 nvidia/linux-aarch64/libnvfatbin-dev-12.5.39-0 nvidia/linux-aarch64/libnvjitlink-12.5.40-0 nvidia/linux-aarch64/libnvjitlink-dev-12.5.40-0 nvidia/linux-aarch64/libnvjpeg-12.3.2.38-0 nvidia/linux-aarch64/libnvjpeg-dev-12.3.2.38-0 main/linux-aarch64/libpng-1.6.39-h998d150_0 main/linux-aarch64/libstdcxx-devel_linux-aarch64-11.2.0-h1234567_1 main/linux-aarch64/libstdcxx-ng-11.2.0-h1234567_1 main/linux-aarch64/libuuid-1.41.5-h998d150_0 main/linux-aarch64/libxcb-1.15-h2f4d8fa_0 main/linux-aarch64/libxkbcommon-1.0.1-h998d150_1 main/linux-aarch64/libxml2-2.10.4-h6097fa9_2 main/linux-aarch64/ncurses-6.4-h419075a_0 nvidia/linux-aarch64/nsight-compute-2024.2.0.16-2 main/linux-aarch64/nspr-4.35-h419075a_0 main/linux-aarch64/nss-3.89.1-h419075a_0 main/linux-aarch64/_openmp_mutex-5.1-51_gnu main/linux-aarch64/openssl-3.0.13-h2f4d8fa_2 main/linux-aarch64/pcre2-10.42-hcfaa891_1 main/linux-aarch64/pip-24.0-py312hd43f75c_0 main/linux-aarch64/python-3.12.3-h8edadfe_1 main/linux-aarch64/readline-8.2-h998d150_0 main/linux-aarch64/setuptools-69.5.1-py312hd43f75c_0 main/linux-aarch64/sqlite-3.45.3-h998d150_0 main/noarch/sysroot_linux-aarch64-2.26-h4744778_14 main/linux-aarch64/tk-8.6.14-h987d8db_0 main/noarch/tzdata-2024a-h04d1e81_0 main/linux-aarch64/wheel-0.43.0-py312hd43f75c_0 main/linux-aarch64/xz-5.4.6-h998d150_1 main/linux-aarch64/zlib-1.2.13-h998d150_1 ```
jakirkham commented 2 months ago

Yes am still seeing it unfortunately as is a colleague. Reproduced just before writing this

We tried a bunch of things:

Always the same issue with creation

jakirkham commented 2 months ago

Should add I wanted to confirm it wasn't something with corp networking. Have reproduced the same error on my personal laptop (which is x86_64 based)

jakirkham commented 2 months ago

Should add the same issue is not present with CUDA 12.4. So this works

conda create -c nvidia -n cuda12.4 cuda=12.4 --dry-run

Mentioning in case this provides any clues as to the differences between CUDA 12.4 & 12.5 packages

jakirkham commented 2 months ago

So I just tried downloading the repodata itself and am not seeing the packages there either

Commands:

curl -L https://conda.anaconda.org/nvidia/linux-aarch64/repodata.json > nvidia_linux-aarch64_repodata.json
cat nvidia_linux-aarch64_repodata.json | grep '12\.5\.0'

Output:

    "cuda-python-12.5.0-py310hdae0e22_0.tar.bz2": {
      "version": "12.5.0"
    "cuda-python-12.5.0-py311hdae0e22_0.tar.bz2": {
      "version": "12.5.0"
    "cuda-python-12.5.0-py312hdae0e22_0.tar.bz2": {
      "version": "12.5.0"
    "cuda-python-12.5.0-py39hdae0e22_0.tar.bz2": {
      "version": "12.5.0"

For some reason only cuda-python shows up, which was uploaded before any of the CTK packages were uploaded

chenghlee commented 2 months ago

Can you try an anaconda login (if you have an anaconda.org) and then attempt the conda create command again?

chenghlee commented 2 months ago

This has been escalated to Anaconda's infrastructure team. We have identified two possible causes and are actively investigating them.

jakirkham commented 1 month ago

Thanks Cheng! 🙏

Hope everyone had a good weekend!

Am curious how things are going here

chenghlee commented 1 month ago

It looks like the channel cloning CDN is having problems dealing with .conda format files on the nvidia channel. We're going to deploy an attempted fix in the morning EU time.

jakirkham commented 1 month ago

Ah ok. That makes sense. This is our first release using .conda

Thanks Cheng! 🙏

Please let us know when we should recheck

jakirkham commented 1 month ago

Did this fix get deployed?

chenghlee commented 1 month ago

We tried it and it didn't work. 😞 We have an alternative approach, but that will take longer to roll out. (Still working on getting an estimate.)

In the meantime, you might want to consider falling back to the older .tar.bz2 format, at least for this release until we get the CDN sorted out.

jakirkham commented 1 month ago

Gotcha, thanks Cheng! 🙏

If we uploaded .tar.bz2 and kept .conda packages, would that work or would that cause issues? IOW should we delete the .conda packages and then upload .tar.bz2 packages?

chenghlee commented 1 month ago

You can upload both formats (i.e., keep the .conda ones already there). The .org backend definitely supports that, and I don't think having both formats would cause any problems since bug in the CDN cloning process is [essentially] just ignoring the repodata "packages.conda" entries supplied by the backend.

BTW, the reason -c nvidia/label/main made the packages appear is that adding labels to the channel name bypasses the CDN clones and connects directly to the .org backend.

adibbley commented 1 month ago

We reuploaded the CUDA packages in .tar.bz2 format and things seem to be working as expected. Thanks!

jakirkham commented 1 month ago

Thanks Cheng and Alex! 🙏

Have rewritten the OP and title to point out the remaining issue of mirroring nvidia channel .conda packages to CDN

jakirkham commented 3 weeks ago

Am curious what the current status is on this issue. It would be nice to have .conda packages for future CUDA releases

Expect users would appreciate the smaller packages for downloads

Not to mention having access to package metadata without needing to do a full decompression is more noticeable with these larger packages

jakirkham commented 2 weeks ago

This happened again with CUDA 12.5.1. We used .tar.bz2