prefix-dev / pixi

Package management made easy
https://pixi.sh
BSD 3-Clause "New" or "Revised" License
2.19k stars 132 forks source link

Allow adding a package from a non-main channel without explicitly adding its dependencies #1532

Open Yura52 opened 1 week ago

Yura52 commented 1 week ago

Problem description

The problematic pixi.toml that cannot be successfully resolved with pixi=0.24.2:

[project]
name = "pixi-faiss"
version = "0.1.0"
description = "Add a short description here"
authors = ["John Doe"]
channels = ["conda-forge", "pytorch"]
platforms = ["osx-arm64"]

[dependencies]
python = ">=3.11.9,<3.12"
faiss-cpu = { version = ">=1.8.0,<2", channel = "pytorch" }
numpy = ">=1.26.4,<2"

The error message:

  × failed to solve the conda requirements of 'default' 'osx-arm64'
  ╰─▶ Cannot solve the request because of: The following packages are incompatible
      └─ faiss-cpu >=1.8.0,<2 cannot be installed because there are no viable options:
         └─ faiss-cpu 1.8.0 | 1.8.0 | 1.8.0 | 1.8.0 would require
            └─ libfaiss ==1.8.0 hcb8d3e5_0_cpu, which cannot be installed because there are no viable
      options:
               └─ libfaiss 1.8.0 is excluded because due to strict channel priority not using this
      option from: 'https://conda.anaconda.org/pytorch/'

It seems that the problem is the libfaiss package (a dependency of faiss-cpu) that is presented in both allowed channels. Adding libfaiss = { channel = "pytorch" } to the dependencies section fixes the issue, but explicitly listing indirect dependencies of a project seems to go against the spirit of dependencies.

So it would be nice not to have to deal with indirect dependencies. That said, it is unclear what should be the default behavior. In the above example, I definitely would like to get libfaiss from the pytorch channel, but I am not sure if this is a representative example.

As a quick fix, in the error message, pixi could explain the situation and offer possible fixes (I found the fix by trial and error).

wolfv commented 1 week ago

We are using strict channel priority. That means, any package from teh first channel "wins" against any package from channels with lower priority.

Things might also work if you put the pytorch channel first?

Yura52 commented 1 week ago

Things might also work if you put the pytorch channel first?

It works, but it changes the default channel for all packages in the project, and I would like to keep conda-forge as the default channel. Basically, my case is similar to the official example, with the difference being that my package from the pytorch channel also has a dependency from that channel.

So it seems that situation is as follows: "if a package from a non-main channel has a dependency that is presented in both main and non-main channels, then a user must explicitly provide a channel for this indirect dependency". Generally, I understand the motivation to ask users to be explicit in such ambiguous cases. Then, a better error message with an actionable advice would be very helpful in such cases.

freundTech commented 3 hours ago

I've run into the same problem trying to install rust-nightly:

 [project]
 name = "test"
 channels = ["conda-forge", "conda-forge/label/rust_dev"]
 platforms = ["linux-64"]

 [tasks]

 [dependencies]
 rust = { version = "==1.81.0.dev20240625", channel = "conda-forge/label/rust_dev" }
  × failed to solve the conda requirements of 'default' 'linux-64'
  ╰─▶ Cannot solve the request because of: The following packages are incompatible
      └─ rust ==1.81.0.dev20240625 cannot be installed because there are no viable options:
         └─ rust 1.81.0.dev20240625 would require
            └─ rust-std-x86_64-unknown-linux-gnu ==1.81.0.dev20240625 h16879b0_0, which cannot be installed because there are no viable options:
               └─ rust-std-x86_64-unknown-linux-gnu 1.81.0.dev20240625 is excluded because due to strict channel priority not using this option from: 'https://conda.anaconda.org/conda-forge/label/rust_dev/'

In this case manually adding the dependencies and specifying the channel isn't practical, as it would have to be done for every platform manually.