Mismatching virtual package error should hint to solution (set `system-requirements`)

pavelzw commented 1 year ago

Problem description

$ pixi init
$ pixi add python
$ pixi install
x The current system has a mismatching virtual package. The project requires '__linux' to be at least version '5.10' but the system has version '4.18.0'

should be something like

$ pixi init
$ pixi add python
$ pixi install
x The current system has a mismatching virtual package. The project requires '__linux' to be at least version '5.10' but the system has version '4.18.0'. Try setting the following in your pixi.toml:
[system-requirements]
linux="4.18"

Similar for other virtual packages.

jpursell commented 6 months ago

This should be re-opened. I just got this unhelpful error

bash-4.4$ pixi shell
  × The current platform does not satisfy the minimal virtual package requirements
  ╰─▶ the project does not support 'linux-64'

pixi version 0.20.1

pavelzw commented 6 months ago

@ruben-arts can you reopen please? I don't have permissions to do so

jpursell commented 6 months ago

I should mention that the fix for this was not to add something to [system-requirements], but rather I needed to add "linux-64" to the platforms. It be very helpful if pixi would tell the user that.

roaldarbol commented 6 months ago

I also just got this for a pypi install:

❯ pixi add --pypi syncstart==1.1.1
  × failed to solve the pypi requirements of 'default' 'osx-64'
  ├─▶ Failed to build: `opencv-python==4.9.0.80`
  ├─▶ Failed to install requirements from build-system.requires (install)
  ├─▶ Failed to download and build distributions
  ├─▶ Failed to fetch wheel: numpy==1.22.2
  ├─▶ Failed to build: `numpy==1.22.2`
  ╰─▶ Build backend failed to build wheel through `build_wheel()` with exit status: 1

... along with the longer stdout stuff. It was resolved by setting:


[system-requirements]
macos = "11.0"

ruben-arts commented 6 months ago

Honestly the users should feel the power of system requirements and not the downside. I've got a proposal to solve this issue with some workflow improvements:

TL;DR: Make pixi more flexible and let it interact with the dependency resolution automatically, and help the user by doing the work for them.

Definitions

Virtual packages: a dependency of a package that can not be provided as a conda package, these are system specifications or libc implementation on the system. e.g. cuda, linux, macos, glibc etc. System requirements: Pixi's take on defining a virtual package in the project's manifest. It defines the requirements the project has of the machine it needs to run on, which it can't provide itself. There is a set of defaults per platform, these are used during the solve of the dependencies to avoid the solve to only work on your specific machine. If the project requires a different set then the default they need to be defined in the [system-requirements] table.

Solution

Let's make pixi solve with the current available virtual packages. Then after the solve figure out what virtual packages are required. If those differ from the default virtual packages they will automatically be added to the manifest file as system requirements. This goes both ways, having higher or lower versions on your current machine.

Solution to newly created problem

The down side of change would be that the solution(lock file) might only work for the latest machines as pixi picks the latest versions available for the given virtual packages. Potentially lowering the amount of machines your project can run on. This could be fixed by giving pixi an interactive option to re-solve for your machine, skipping the system-requirements given by the project. When it finds a different solution for your machine it automatically writes it back to the manifest, allowing the users to track these changes in vcs.

Example of the interactive prompt:

> pixi install
The project requires your system to have `cuda 12.0`. 
Should I try to solve a new environment with this machine's specs?
 > Yes
 - No

Edit: After some discussion about the solution to this issue. Here is a more specific resulting situation:

$ pixi info
...
  Virtual packages: __cuda=12.5=0
...
$ pixi add pytorch

[system-requirements]
# Add system requirements that are non default
cuda = "12.1"

Change I expect to be needed:

Pixi needs to share the machines virtual packages instead of the projects requirements with the solver.
Pixi needs to figure out the version of the virtual packages from the locked environment.
- For conda packages it easy to just look add the dependencies of all the repodata records
- For pypi the information can be read from the wheel names. Still have to figure out which wheel to select but the first version should have the same behavior as of right now.

ping @baszalmstra @wolfv for input

roaldarbol commented 6 months ago

I think it IS really powerful, so I hope it will become super intuitive! For lots of tracking software (e.g. https://github.com/DeepLabCut/DeepLabCut, https://gitlab.com/polavieja_lab/idtrackerai) installation has been a major sticking point for adoption, especially since they depend on TensorFlow and thus getting cuda just right across platforms - this would be greatly improved by a simple specification.

For me personally, I think one of the confusing aspects have been around the OS-named system-requirements, and how they differ from platforms. E.g. if I have already specified osx-64 as the platform, it's been unintuitive why I'd need to additionally specify macos = "11.0" under system-requirements. Usually, the version requirements is to narrow the solving, so it took me by surprise that you could get something to solve by specifying a version - I'd think it would otherwise just attempt to solve against all system requirements for the specified platform.

Lastly, it would be great if pixi could "figure out" the correct requirement for the user's current platform. E.g. for cuda, It would make life easier for users if they didn't have to know the CUDA requirements for their particular platform. Here the target audience are not super tech-savvy, but would like to use techy tools easily. I guess that's part of @ruben-arts's proposal - is that correctly understood?

baszalmstra commented 6 months ago

I think your idea can work @ruben-arts but I cant really see the ramifications yet. Lets try to build an experiment in the near future.

ruben-arts commented 6 months ago

Hey @roaldarbol, Thanks for the input, this is good to take into account in the UX design. I believe that the things you mention are all atleast what I tried to explain in my proposal. So lets see what we can come up with in the first version!

zachcp commented 6 months ago

I just ran into this as well and wasn't sure what it meant.

$ pixi run test
Pixi task (test in default): pytest tests
  × The current system has a mismatching virtual package. The project requires
  │ '__linux' to be at least version '5.10' but the system has version '5.4.0'

ruben-arts commented 6 months ago

@zachcp Yep exact same issue.

Try:

# [tool.pixi.system-requirements] for pyproject.toml
[system-requirements]
linux = "5.4"

prefix-dev / pixi