pixi does not respect channel and mirrors settings

[ianzone]

Checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

~/.pixi/config.toml

detached-environments = false

[mirrors]
"https://conda.anaconda.org/" = ["https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/"]
"https://conda.anaconda.org/nvidia" = ["https://mirrors.sustech.edu.cn/anaconda-extra/cloud/nvidia/"]

[repodata-config]
disable-jlap = true
disable-zstd = true

pixi.toml

[project]
channels = ["nvidia", "conda-forge", "pytorch"]
name = "d2l"
platforms = ["linux-64"]
version = "0.1.0"

[dependencies]
cuda = { channel = "nvidia" }

❯ pixi search cuda

Using channels: nvidia, conda-forge, pytorch

cuda ha804496_0
---------------

Name                cuda
Version             12.6.1
Build               ha804496_0
Size                26689
License             LicenseRef-NVIDIA-End-User-License-Agreement
Subdir              noarch
File Name           cuda-12.6.1-ha804496_0.conda
URL                 https://conda.anaconda.org/conda-forge/noarch/cuda-12.6.1-ha804496_0.conda
MD5                 79c7f0280c94d5b9707fd1fcf4832738
SHA256              11bc3c354b01be0b10d0a354ec91e6e847e8af2f798bbe6eaced12c07235f4c6

Dependencies:
 - __linux
 - cuda-runtime 12.6.1.*
 - cuda-toolkit 12.6.1.*

Issue description

the cuda info is fetched from neither nvidia channel nor the mirrors.tuna.tsinghua.edu.cn

Expected behavior

the URL should be like https://mirrors.sustech.edu.cn/anaconda-extra/cloud/nvidia/cuda-12.6.1-ha804496_0.conda

[deltamarnix]

I am trying to do the same thing with a JFrog Artifactory.

This is my C:\Users\<user>\.pixi\config.toml:

[mirrors]
"https://conda.anaconda.org/" = [
    "<EMAIL>:<TOKEN>@<COMPANY>.jfrog.io/artifactory/api/conda/<COMPANY>-conda"
]

This is the command: pixi search python. The results seem to use the default anaconda.org/conda-forge repo.

Using channels: conda-forge

python hf5aa216_100_cp313

Name python Version 3.13.0 Build hf5aa216_100_cp313 Size 16641177 License Python-2.0 Subdir win-64 File Name python-3.13.0-hf5aa216_100_cp313.conda URL https://conda.anaconda.org/conda-forge/win-64/python-3.13.0-hf5aa216_100_cp313.conda MD5 3c2f7ad3f598480fe2a09e4e33cb1a2a SHA256 18f3f0bd514c9101d38d57835b2d027958f3ae4b3b65c22d187a857aa26b3a08

Dependencies:

• bzip2 >=1.0.8,<2.0a0
• libexpat >=2.6.3,<3.0a0
• libffi >=3.4,<4.0a0
• libmpdec >=4.0.0,<5.0a0
• libsqlite >=3.46.1,<4.0a0
• libzlib >=1.3.1,<2.0a0
• openssl >=3.3.2,<4.0a0
• python_abi 3.13. _cp313
• tk >=8.6.13,<8.7.0a0
• tzdata
• ucrt >=10.0.20348.0
• vc >=14.2,<15
• vc14_runtime >=14.29.30139
• xz >=5.2.6,<6.0a0

[deltamarnix]

In addition I want to mention that using default-channels does seem to work:

default-channels = ["<EMAIL>:<TOKEN>@<COMPANY>.jfrog.io/artifactory/api/conda/<COMPANY>-conda"]

Using -vvv for extra debugging options show that default-channels is used when performing the search (outside of a project folder).

Using -vvv on my original mirror I got the following information:

ERROR get_or_create_subdir{channel=Channel { platforms: None, base_url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("conda.anaconda.org")), port: None, path: "/conda-forge/", query: None, fragment: None }, name: Some("conda-forge") } platform=NoArch}:fetch_repo_data{cache_path=C:\Users\<user>\AppData\Local\rattler/cache\repodata}: rattler_repodata_gateway::fetch: error=HTTP status client error (401 Unauthorized) for url (https://<email>:********@<company>.jfrog.io/artifactory/api/conda/<company>-conda/conda-forge/noarch/repodata.json.zst)
ERROR get_or_create_subdir{channel=Channel { platforms: None, base_url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("conda.anaconda.org")), port: None, path: "/conda-forge/", query: None, fragment: None }, name: Some("conda-forge") } platform=NoArch}: rattler_repodata_gateway::gateway: error=HTTP status client error (401 Unauthorized) for url (https://<email>:********@<company>.jfrog.io/artifactory/api/conda/<company>-conda/conda-forge/noarch/repodata.json.zst)
  x HTTP status client error (401 Unauthorized) for url (https://<email>:********@<company>.jfrog.io/artifactory/api/conda/<company>-conda/conda-forge/noarch/repodata.json.zst)

[wolfv]

Hi @ianzone it's by "design" that the URL that is reported in the lockfile or in the search is reported as the original URL. A mirror is just a "mirror" of the original and is a "user" setting. For compatibility with others, the official URL is retained.

You can test that the configuration works by setting it to an invalid URL, such as https://foobar.dev. Then you will see errors like the following:

╭─wolfv@pixi ~/Programs/shell ‹main●›
╰─$ bat ~/.pixi/config.toml                                                                                                                                                                                                                    1 ↵
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /Users/wolfv/.pixi/config.toml
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ [mirrors]
   2   │ "https://conda.anaconda.org" = [
   3   │     "https://foobar.dev/"
   4   │ ]
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
╭─wolfv@pixi ~/Programs/shell ‹main●›
╰─$ pixi search foo
Using channels: conda-forge
ERROR error=repodata not found
ERROR error=repodata not found
ERROR error=could not find subdir 'noarch' in channel 'https://conda.anaconda.org/conda-forge/'
  × could not find subdir 'noarch' in channel 'https://conda.anaconda.org/conda-forge/'
  ╰─▶ HTTP status client error (404 Not Found) for url (https://www.foobar.dev/conda-forge/noarch/repodata.json.zst)

Does this make sense?

[deltamarnix]

Thanks @wolfv , that seems to work for me. I still get a 401 Unauthorized. I will check with JFrog if it's on their side.

[wolfv]

@deltamarnix do you know if the .zst repodata is available on Artifactory? Maybe they are buggy and return a 401 but the real problem is that this specific file is not available at all.

[wolfv]

@deltamarnix you can try to disable zstd & bzip2 to see if that changes anything. I have heard about buggy artifactories before: https://pixi.sh/latest/reference/pixi_configuration/#repodata-config

[deltamarnix]

I disabled zstd and bzip, then it tries to get the json and I receive the same error for json. And the strange thing is that it works directly in the browser.

This is the error from pixi: HTTP status client error (401 Unauthorized) for url (https://name>%40<company>.nl:********@<company.jfrog.io/artifactory/api/conda/-conda/conda-forge/noarch/repodata.json)

And the json that I get when using the url directly in the browser (I cannot share the real url, because of the secret token). I used a private session to make sure that there are no cookies or sessions.

{
  "info" : {
    "subdir" : "noarch"
  },
  "packages" : { },
  "packages.conda" : { },
  "removed" : [ ]
}

My config:

[mirrors]
"https://conda.anaconda.org" = [
    "https://<name>%40<company>.nl:<token>@<company>.jfrog.io/artifactory/api/conda/<company>-conda"
]

[repodata-config]
disable-bzip2 = true
disable-zstd = true

Full error with pixi search foo -vvv

ERROR get_or_create_subdir{channel=Channel { platforms: None, base_url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("conda.anaconda.org")), port: None, path: "/conda-forge/", query: None, fragment: None }, name: Some("conda-forge") } platform=NoArch}:fetch_repo_data{cache_path=C:\Users\<user>\AppData\Local\rattler/cache\repodata}: rattler_repodata_gateway::fetch: error=HTTP status client error (401 Unauthorized) for url (https://<name>%40<company>.nl:********@<company>.jfrog.io/artifactory/api/conda/<company>-conda/conda-forge/noarch/repodata.json)
ERROR get_or_create_subdir{channel=Channel { platforms: None, base_url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("conda.anaconda.org")), port: None, path: "/conda-forge/", query: None, fragment: None }, name: Some("conda-forge") } platform=NoArch}: rattler_repodata_gateway::gateway: error=HTTP status client error (401 Unauthorized) for url (https://<name>%40<company>.nl:********@<company>.jfrog.io/artifactory/api/conda/<company>-conda/conda-forge/noarch/repodata.json)

[wolfv]

The last thing you could try is remove the password / token and use pixi auth for the URL.

That would look like: pixi auth login <company>.jfrog.io/artifactory --user <company> --password <name>%40<company>.nl

Then you can use https://<company>.jfrog.io/artifactory/api/conda/<company>-conda and maybe things will work?!

If that works, I'll have a look at our way of dealing with password URLs.

[deltamarnix]

pixi auth login worked for me and I'm up for new errors. Sorry for hijacking this issue. Perhaps I should make a new issue now.

[mirrors]
"https://conda.anaconda.org" = [
    "https://<company>.jfrog.io/artifactory/api/conda/<company>-conda"
]

pixi auth login --username <email> --password <token> https://<company>.jfrog.io/artifactory pixi search python

Error:

Using channels: conda-forge
ERROR error=repodata not found
  x Package python not found, please use a wildcard '*' in the search name for a broader result.

Maybe this has to do with the fact that jfrog is a proxy and not a mirror?

[wolfv]

Hmm, this would be only related to what pixi finds in the repodata.json file. It's hard to say. Maybe you can look at the files in the cache (pixi info shows you the cache location) and then there are a bunch of JSON files. Or run with -v to see the actual URLs...

[deltamarnix]

These are the json files in my cache.

36521c4e.json

{
  "info" : {
    "subdir" : "noarch"
  },
  "packages" : { },
  "packages.conda" : { },
  "removed" : [ ]
}

36521c4e.info.json

{
  "url": "https://conda.anaconda.org/conda-forge/noarch/repodata.json.zst",
  "etag": "ad8c89902396f41688f42b26459860ca38c18801",
  "mod": "Thu, 10 Oct 2024 07:54:46 GMT",
  "cache_control": "no-store",
  "mtime_ns": 1728553930998210800,
  "size": 109,
  "blake2_hash": "701763ce7b4ca8cbee5c2f97c9b599ecb4a97f221fd9ebc0e45eb001d13a5527",
  "blake2_hash_nominal": "701763ce7b4ca8cbee5c2f97c9b599ecb4a97f221fd9ebc0e45eb001d13a5527",
  "has_zst": {
    "value": true,
    "last_checked": "2024-10-10T09:52:10.978299600Z"
  },
  "has_bz2": {
    "value": true,
    "last_checked": "2024-10-10T09:52:10.978131100Z"
  },
  "has_jlap": {
    "value": false,
    "last_checked": "2024-10-10T09:52:10.978307400Z"
  },
  "jlap": null
}

[wolfv]

This hints at downlaoding a wrong (empty!!) file.

Pixi (conda, mamba) accept a channel as working when a noarch/repodata.json file can be found. It looks like it did find a noarch/repodata.json file in your case, but no data inside and no corresponding linux-64/repodata.json file (or whatever arch you are on).

So I think something is still not right with your configuration.

I would analyze the full URL that pixi is requesting.

[deltamarnix]

I have come further with my investigations and have come to the conclusion that the URL buildup is wrong for me when using mirrors.

My config.toml:

[mirrors]
"https://conda.anaconda.org/" = ["https://<company>.jfrog.io/artifactory/api/conda/<company>-conda-remote"]

And I used pixi auth login --token <specialtokenfromjfrog> https://<company.jfrog.io to log in.

I get the following error:

ERROR error=could not find subdir 'noarch' in channel 'https://conda.anaconda.org/conda-forge/'
  × could not find subdir 'noarch' in channel 'https://conda.anaconda.org/conda-forge/'
  ╰─▶ HTTP status client error (404 Not Found) for url (https://<company>.jfrog.io/artifactory/api/conda/<company>-
      conda-remote/conda-forge/noarch/repodata.json)

And now it makes sense to me, because this file does not exist. The file that does exist is on the link: https://<company>.jfrog.io/artifactory/api/conda/<company>-conda-remote/noarch/repodata.json. So the part with "conda-forge" has to be removed from the link. Are there options within pixi on how to perform the mirror link replacement?