prefix-dev / pixi

Package management made easy
https://pixi.sh
BSD 3-Clause "New" or "Revised" License
3.32k stars 185 forks source link

Issue installing `numba` and `pytorch` together. #2403

Open Orenbek opened 1 week ago

Orenbek commented 1 week ago

Checks

Reproducible example

  1. copy paste the following content to pixi.toml then run pixi install.
    
    [project]
    channels = ["conda-forge", "pytorch"]
    platforms = ["osx-arm64"]
    version = "0.1.0"

[tasks]

[dependencies] python = "3.11." pytorch = {version = "2.5.", channel="pytorch"}

2. run `pixi add numba`

### Issue description

I encountered an issue using Pixi to manage dependencies on macOS (osx-arm64) in a Python 3.11 environment. When trying to add the numba package with pixi add numba, Pixi fails to find a compatible version and returns a version mismatch error.
Here's the specific error message:
```shell
× failed to solve the conda requirements of 'default' 'osx-arm64'
  ╰─▶ Cannot solve the request because of: The following packages are incompatible
      ├─ numba * can be installed with any of the following options:
      │  ├─ numba 0.55.0 | 0.55.1 | 0.55.1 | 0.55.2 | 0.56.2 | 0.56.2 | 0.56.3 | 0.56.4 | 0.56.4 | 0.57.0 | 0.57.0 | 0.57.0 | 0.57.1 |
      0.58.1 | 0.58.1 | 0.59.0 | 0.59.0 | 0.59.1 | 0.60.0 would require
      │  │  └─ python_abi 3.10.* *_cp310, which can be installed with any of the following options:
      │  │     └─ python_abi 3.10
      │  ├─ numba 0.53.0 | 0.53.0 | 0.53.1 | 0.53.1 | 0.54.0 | 0.54.1 | 0.55.0 | 0.55.1 | 0.55.1 | 0.55.2 | 0.56.2 | 0.56.2 | 0.56.3 |
      0.56.4 | 0.56.4 | 0.57.0 | 0.57.0 | 0.57.0 | 0.57.1 | 0.58.1 | 0.58.1 | 0.59.0 | 0.59.0 | 0.59.1 | 0.60.0 would require
      │  │  └─ python_abi 3.9.* *_cp39, which can be installed with any of the following options:
      │  │     └─ python_abi 3.9 | 3.9
      │  └─ numba 0.52.0 | 0.53.0 | 0.53.0 | 0.53.1 | 0.53.1 | 0.54.0 | 0.54.1 | 0.55.0 | 0.55.1 | 0.55.1 | 0.55.2 | 0.56.2 | 0.56.2 |
      0.56.3 | 0.56.4 | 0.56.4 | 0.57.0 | 0.57.0 | 0.57.0 | 0.57.1 | 0.58.1 | 0.58.1 would require
      │     └─ python_abi 3.8.* *_cp38, which can be installed with any of the following options:
      │        └─ python_abi 3.8 | 3.8

Expected behavior

Upon further investigation, I found that the issue is due to Pixi not selecting the correct build hash for Python 3.11, even though numba already supports this version. For example, a conda search numba=0.60.0 shows:

# Name                       Version           Build  Channel             
numba                         0.60.0 py310h0628f0e_0  conda-forge         
numba                         0.60.0 py311h9506ed5_0  conda-forge         
numba                         0.60.0 py312h41cea2d_0  conda-forge         
numba                         0.60.0  py39h2d4ef1e_0  conda-forge

Pixi should ideally select py311h9506ed5_0 for compatibility with Python 3.11, which would prevent the mismatch error.

ruben-arts commented 1 week ago

I've tried the issue, but there seems to be some really strict dependencies between these package. Did you manage to install it on osx-arm64 before with something other than pixi? It can solve for linux-64.

Edit: Found some more info, pytorch requires llvm-openmp <16 and numba requires llvm-openmp >=16.0.4 or more.

So on osx-arm64 they are not able to install together.

ruben-arts commented 1 week ago

Ah it works if you don't use the pytorch channel.

[project]
channels = ["conda-forge"]
platforms = ["osx-arm64"]
version = "0.1.0"
name = "numba"

[dependencies]
python = "3.11.*"
numba = "*"
pytorch = "2.5.*"
Orenbek commented 1 week ago

I'll try it out later. Thanks!