search

prefix-dev / pixi

Package management made easy
https://pixi.sh
BSD 3-Clause "New" or "Revised" License
3.28k stars 181 forks source link

Tracking issue: PyPI dependencies pixi can't manage yet. #771

Open ruben-arts opened 9 months ago

ruben-arts commented 9 months ago

There are a few PyPI packages pixi can't install yet, where pip can.

Please paste your examples in this issue so we have a list of known packages we can track, test and benchmark with along the way.

Information we would like:

  1. What did you run and what was the outcome? e.g. pixi add --pypi packagex If it doesn't recreate in an empty environment please share your pixi.toml that recreates the issue.

  2. What error did pixi return? e.g. 

    × RECORD file doesn't match wheel contents: missing hash for mediapipe/version.txt (expected sha256=-fE2KU)

  3. Can pip install the package? Does pip install packagex work?

  4. What platform are you on? e.g. linux-64

  5. Did you find a workaround, if so please explain. e.g. build it into a conda package, using a custom fork, etc.

Your input would greatly help us improve the pixi's experience! Thanks in advance! :heart:

tdejager commented 2 months ago

@liblaf I've upgraded your comment into an issue so that its easier to track and make all of those work. See: #1926

urucoder commented 2 months ago

~@urucoder in your case I would try glibc 2.17 I think.~

nvidia-cuda-nvrtc-cu11==11.8.89 works with glibc 2.35 with pip and uv

liblaf commented 2 months ago

~@urucoder in your case I would try glibc 2.17 I think.~

nvidia-cuda-nvrtc-cu11==11.8.89 works with glibc 2.35 with pip and uv

@urucoder should be fixed by #1925, tested here

evetion commented 1 month ago

Not sure if this is the right issue, but in a git checkout, while #subdirectory is parsed, it can't be cloned:

pixi add --pypi -f dev "ribasim@git+https://github.com/Deltares/Ribasim.git#subdirectory=python/ribasim" --editable
 WARN Ignoring editable flag for non-path requirements.
 WARN Ignoring editable flag for non-path requirements.
  ⠤ dev:osx-64           [00:00:01] cloning https://github.com/Deltares/Ribasim.git#subdirectory=python/ribasim
  × failed to solve the pypi requirements of 'dev' 'osx-64'
  ├─▶ failed to resolve pypi dependencies
  ├─▶ Failed to download and build `ribasim @ git+https://github.com/Deltares/Ribasim.git#subdirectory=python/ribasim#subdirectory=python/ribasim`
  ├─▶ Git operation failed
  ├─▶ failed to fetch into: C:\Users\pronk_mn\AppData\Local\rattler/cache\uv-cache\git-v0\db\3e8b602b670cc8d6
  ╰─▶ process didn't exit successfully: `git fetch --force --update-head-ok https://github.com/Deltares/Ribasim.git#subdirectory=python/ribasim +HEAD:refs/remotes/origin/HEAD`
      (exit code: 128)
      --- stderr
      fatal: unable to update url base from redirection:
        asked for: https://github.com/Deltares/Ribasim.git#subdirectory=python/ribasim/info/refs&service=git-upload-pack
         redirect: https://github.com/Deltares/Ribasim

It seems the url fragment should be stripped when cloning. Also note that the url fragment is duplicated on the line starting with Failed to download and build.

floringogianu commented 1 month ago

I had issues with two jax packages:

In both cases, on osx-arm64,

pixi add equinox

errors with:

thread 'tokio-runtime-worker' panicked at library/core/src/slice/sort/shared/smallsort.rs:862:5:
user-provided comparison function does not correctly implement a total order

Fixed it with pixi add --pypi ....

Project config:

[project]
authors = [{name = "...", email = "..."}]
dependencies = []
description = "Add a short description here"
name = "lru"
requires-python = ">= 3.12"
version = "0.1.0"

[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["osx-arm64"]

[tool.pixi.pypi-dependencies]
lru = { path = ".", editable = true }

[tool.pixi.dependencies]
jax = ">=0.4.30"
pytorch = ">=2.4.0"
ruff = "*"
ipython = "*"

# Feature Definitions
[tool.pixi.feature.cuda]
platforms = ["linux-64"]
system-requirements = {cuda = "12"}

[tool.pixi.feature.cuda.dependencies]
jaxlib = { version = "*", build = "*cuda12*" }

# Environments
[tool.pixi.environments]
cuda = ["cuda"]
ruben-arts commented 1 month ago

@floringogianu I'm unable to reproduce your issue. Is it a recurring issue? Or did you only see it one time?

floringogianu commented 2 weeks ago

@ruben-arts got around to test this again and now I'm not seeing this error anymore 🤷.

traversaro commented 2 weeks ago

On pixi 0.34.0, with drake (https://github.com/RobotLocomotion/drake) we have this problem:

traversaro@IITBMP014LW012:~/drakepixi$ rm -rf *
traversaro@IITBMP014LW012:~/drakepixi$ pixi init .
✔ Created /home/traversaro/drakepixi/pixi.toml
traversaro@IITBMP014LW012:~/drakepixi$ pixi add python==3.11.* pip
✔ Added python==3.11.*
✔ Added pip >=24.2,<25
traversaro@IITBMP014LW012:~/drakepixi$ pixi add --pypi drake
  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ╰─▶ Because only drake>=0.35.0 is available and drake>=0.35.0,<=1.21.0 has no wheels with a matching Python ABI tag,
      we can conclude that drake<0.36.0 cannot be used.
      And because drake>=1.22.0 has no wheels with a matching Python implementation tag and you require drake, we can
      conclude that your requirements are unsatisfiable.

      hint: Pre-releases are available for drake in the requested range (e.g., 0.35.0b1), but pre-releases weren't
      enabled (try: `--prerelease=allow`)
traversaro@IITBMP014LW012:~/drakepixi$ pixi info
System
------------
      Pixi version: 0.34.0
          Platform: linux-64
  Virtual packages: __unix=0=0
                  : __linux=5.15.153.1=0
                  : __glibc=2.39=0
                  : __cuda=12.3=0
                  : __archspec=1=skylake
         Cache dir: /home/traversaro/.cache/rattler/cache
      Auth storage: /home/traversaro/.rattler/credentials.json
  Config locations: No config files found

Global
------------
           Bin dir: /home/traversaro/.pixi/bin
   Environment dir: /home/traversaro/.pixi/envs
      Manifest dir: /home/traversaro/.pixi/manifests/pixi-global.toml

Project
------------
              Name: drakepixi
           Version: 0.1.0
     Manifest file: /home/traversaro/drakepixi/pixi.toml
      Last updated: 23-10-2024 17:06:51

Environments
------------
       Environment: default
          Features: default
          Channels: conda-forge
  Dependency count: 2
      Dependencies: python, pip
  Target platforms: linux-64

@ruben-arts do you prefer this as a new issue or here? Thanks!

fyi @xela-95

traversaro commented 1 week ago

@ruben-arts do you prefer this as a new issue or here? Thanks!

@ruben-arts no hurry on our side, but just to understand, do you prefer this bug report as a comment here or on another issue, thanks!

ruben-arts commented 1 week ago

@traversaro That problem has to do with the system requirements. I really want to get rid of this issue through https://github.com/prefix-dev/pixi/issues/346

Adding the following to the pixi.toml fixes it:

[system-requirements]
libc = "2.35"

Edit: sometimes it helps to look at https://pypi.org/simple/drake/ available wheels.

traversaro commented 1 week ago

@traversaro That problem has to do with the system requirements. I really want to get rid of this issue through https://github.com/prefix-dev/pixi/issues/346

Adding the following to the pixi.toml fixes it:

[system-requirements]
libc = "2.35"

Edit: sometimes it helps to look at https://pypi.org/simple/drake/ available wheels.

@xela-95

traversaro commented 1 week ago

@traversaro That problem has to do with the system requirements. I really want to get rid of this issue through #346

Adding the following to the pixi.toml fixes it:

[system-requirements]
libc = "2.35"

Edit: sometimes it helps to look at https://pypi.org/simple/drake/ available wheels.

Thanks @ruben-arts, this fixed our problem! Cross-linking https://github.com/prefix-dev/pixi/issues/346 that seems to be the actual root issue here.