Pixi not able to solve pypi packages with compatible release specifier (~=)

[hrz6976]

Checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

pixi add python~=3.10
pixi add multimodal-transformers --pypi
pixi add tqdm

Issue description

I'm trying to migrate a conda project to pixi, and surprised to find that multimodal-transformers and tqdm can not be installed together 😮. From the logs, seems that pixi thinks I'm requiring tqdm==4.66.5 (latest version) when I pixi add tqdm, which conflicts with tqdm~=4.64.1; pip recognizes in this case and installs tqdm==4.64.1.

Output from pixi add tqdm -vvv:

 INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform win-64 are out of date because the requirement 'tqdm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform linux-64 are out of date because the requirement 'tqdm *' could not be satisfied (required by '<environment>')
...
 INFO resolvo::solver: deciding to assign tqdm=4.66.5=pyhd8ed1ab_0, (<root> requires tqdm *, 106 possible candidates)
 INFO resolvo::solver: ╤══ Install tqdm=4.66.5=pyhd8ed1ab_0 at level 149 (required by <root>)
...
DEBUG solve:solve_tracked: uv_resolver::resolver: Adding transitive dependency for multimodal-transformers==0.3.1: tqdm>=4.64.1, <4.65.dev0
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: multimodal-transformers @ 0.3.1    
DEBUG solve:solve_tracked: uv_resolver::resolver: Adding transitive dependency for multimodal-transformers==0.3.1: tqdm==4.66.5
DEBUG solve:solve_tracked: uv_resolver::resolver: Adding transitive dependency for multimodal-transformers==0.3.1: pytest>=7.2.2, <7.3.dev0
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: multimodal-transformers @ 0.3.1    
 INFO solve:solve_tracked: pubgrub::internal::core: Start conflict resolution because incompat satisfied:
   multimodal-transformers ==0.3.1 depends on tqdm >=4.64.1, <4.65.dev0    
 INFO solve:solve_tracked: pubgrub::internal::core: prior cause: multimodal-transformers ==0.3.1 is forbidden    
 INFO solve:solve_tracked: pubgrub::internal::core: backtrack to DecisionLevel(1)    
 INFO solve:solve_tracked: pubgrub::internal::core: Start conflict resolution because incompat satisfied:
   multimodal-transformers ==0.3.1 depends on tqdm >=4.64.1, <4.65.dev0    
DEBUG solve:solve_tracked:choose_version{package=multimodal-transformers}: uv_resolver::resolver: Searching for a compatible version of multimodal-transformers (>0.3.1, <0.4)
DEBUG solve:solve_tracked: uv_resolver::resolver: No compatible version found for: multimodal-transformers

Requirements of multimodal-transformers:

  name: multimodal-transformers
  version: 0.3.1
  url: https://files.pythonhosted.org/packages/5c/80/cd2dad26fdfebdc39cc843f5d62617fdc62140d193163395adf10c4fe600/multimodal_transformers-0.3.1-py3-none-any.whl
  sha256: dd8721efc3c18a7097c93977f6c295b328eb9d8d3fcb7266d6e163b5aea2de45
  requires_dist:
  - transformers>=4.34.1
  - accelerate>=0.24.1
  - torch>=2.0.1
  - sacremoses~=0.0.53
  - networkx~=2.6.3
  - scikit-learn~=1.0.2
  - scipy~=1.11.3
  - pandas~=1.3.5
  - numpy~=1.26.1
  - tqdm~=4.64.1
  - pytest~=7.2.2

Expected behavior

Tqdm can be installed.

[baszalmstra]

This is a shortcoming of how we solve mixed conda and pypi dependencies. We first solve all conda dependencies and use that as input to solve the pypi packages. Conda package are pinned so versions cannot be changed by the pypi solve.

As you already deduced, tqdm is solved by the conda solver to version 4.66.5, but that version is incompatible with the requirement from multimodal-transformers so the solve fails.

Possible solutions are (in order of personal preference):

1 Limit tqdm to the same constraints as multimodal-transformers. e.g. pixi add tqdm~=4.64.1 2 Install tqdm from pypi too. e.g. pixi remove tqdm && pixi add tqdm --pypi 3 Add multimodal-transformers as a conda package by adding it to conda-forge

[hrz6976]

This is a shortcoming of how we solve mixed conda and pypi dependencies. We first solve all conda dependencies and use that as input to solve the pypi packages. Conda package are pinned so versions cannot be changed by the pypi solve.

As you already deduced, tqdm is solved by the conda solver to version 4.66.5, but that version is incompatible with the requirement from multimodal-transformers so the solve fails.

Possible solutions are (in order of personal preference):

1 Limit tqdm to the same constraints as multimodal-transformers. e.g. pixi add tqdm~=4.64.1

2 Install tqdm from pypi too. e.g. pixi remove tqdm && pixi add tqdm --pypi

3 Add multimodal-transformers as a conda package by adding it to conda-forge

Thanks for the timely response 😄! Do you think it will be better to address this limitation and the solution somewhere in the docs so that others won't get confused?

[niemiaszek]

I know it's unrelated to the sole matter of this issue, but can you imagine it being possible to solve pypi and conda packages together in the future? If I remember correctly, that was in the scope with rip, but there was communication that it's not so simple with uv solving pypi.

From my experience, most of the issues with environments is caused by this specific scenario. --frozen gives a lot of stability, so existing envs don't break, but it's still an issue when changing environment or just some packages with "*" versions getting updates. First solving conda -> then pypi failing because of different constraints. On the other hand, often there is very simple quickfix - adding the constraints for the conflicting dependency to conda section, so the 1 solution from your list.

Exactly my issue recently with most packages bumping up to numpy 2.0 and some pypi package still constrained by <2.0. Maybe there would be possibility to automate addition of numpy ~=1.0 to conda dependencies?

Verbosity for the pypi failures is now much better than before, but it's also somehow not concrete. I like the error telling me that e.g. Tensorflow 2.15 requires numpy 2>x>lower and current numpy from conda is 2.0.1, but there is no clear information (without going through all -vvv log) how numpy 2.0.1 even got there in first place without being explicit in manifest. Could be helpful if there was also a list of conda packages with numpy dependency causing this conflict point. This way user could know what packages versions he might need to change manually to support constraints from pypi section if 1 solution wont work.

I know it's very sloppy wording with no direct solution, but for me it seems like there is still a possibility to improve UX with environment setup.

[ruben-arts]

Thanks for your input @niemiaszek, we know of this problem well but have yet to find a good solution to this issue.

That said, even with a cross ecosystem solve solution it is still going to be best to get everything from conda-forge as that is a much more stable distribution and there is guaranteed ABI compatibility. The current message to the community is to help yourself and everyone else by contributing to conda-forge and make the packages you need from pypi a conda package.

[baszalmstra]

@ruben-arts I think the actionable part of this issue is to clarify this in the docs? Or is it already somewhere?