Pypi dependency build fails when conda dependency is needed

jeremyben commented 1 month ago

Checks

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

Trying to install detectron2 from source, which has a build and runtime dependency on pytorch. Haven't got this problem with miniconda.

[project]
channels = ["conda-forge", "pytorch", "main"]
platforms = ["linux-64"]

[dependencies]
python = "3.10.12"
pytorch = { version = "2.4.0", channel = "pytorch" }
pytorch-cuda = { version = "12.4", channel = "pytorch" }
torchvision = { version = "0.19.0", channel = "pytorch" }
numpy = { version = "1.24.4" }

[pypi-dependencies]
detectron2 = { git = "https://github.com/facebookresearch/detectron2", rev = "bcfd464d0c810f0442d91a349c0f6df945467143" }

Issue description

ModuleNotFoundError: No module named 'torch' during detectron2 build step.

⠂ default:linux-64     [00:00:00] resolving pypi dependencies
  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ├─▶ Failed to download and build `detectron2 @ git+https://github.com/facebookresearch/detectron2@bcfd464d0c810f0442d91a349c0f6df945467143`
  ├─▶ Failed to build: `detectron2 @ git+https://github.com/facebookresearch/detectron2@bcfd464d0c810f0442d91a349c0f6df945467143`
  ╰─▶ Build backend failed to determine extra requires with `build_wheel()` with exit status: 1
      --- stdout:

      --- stderr:
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File "/home/jeremy/.cache/rattler/cache/uv-cache/environments-v0/.tmp2nlWj7/lib/python3.10/site-packages/setuptools/build_meta.py", line 327, in
      get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/home/jeremy/.cache/rattler/cache/uv-cache/environments-v0/.tmp2nlWj7/lib/python3.10/site-packages/setuptools/build_meta.py", line 297, in
      _get_build_requires
          self.run_setup()
        File "/home/jeremy/.cache/rattler/cache/uv-cache/environments-v0/.tmp2nlWj7/lib/python3.10/site-packages/setuptools/build_meta.py", line 497, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/home/jeremy/.cache/rattler/cache/uv-cache/environments-v0/.tmp2nlWj7/lib/python3.10/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 10, in <module>
      ModuleNotFoundError: No module named 'torch'
      ---

Expected behavior

Pypi build step should be able to resolve conda installed modules.

ruben-arts commented 1 month ago

I can reproduce this exactly. It seems weird to me that it can't access the conda environment as there import torch does work.

@tdejager We did already activate the conda env right?

tdejager commented 4 weeks ago

Yes, but because we use isolated builds this does not include python dependencies.

ruben-arts commented 4 weeks ago

Right, so the only way for now would be to run:

pixi add uv setuptools

and then run the following task, which you could put in a pixi task.

pixi run uv pip install --no-build-isolation git+https://github.com/facebookresearch/detectron2@bcfd464d0c810f0442d91a349c0f6df945467143

ruben-arts commented 4 weeks ago

To give a little more background for users finding this issue. This issue is something we still have to fix. It's part of the build-isolation story.

detectron2 requires torch but doesn't depend on it.
Pixi uses build-isolation for building pypi-dependencies meaning all "conda-python" dependencies are not taking into account by the pypi build environment.
Thus detectron2 gets build in an empty environment with no way to give it torch.

There are multiple issues explaining this. But it would be best to track https://github.com/prefix-dev/pixi/issues/1124 and please :+1: (upvote) the issue.

I'll close this issue for now as it has become a duplicate.

tdejager commented 4 weeks ago

Right, so the only way for now would be to run:
pixi add uv setuptools
and then run the following task, which you could put in a pixi task.
pixi run uv pip install --no-build-isolation git+https://github.com/facebookresearch/detectron2@bcfd464d0c810f0442d91a349c0f6df945467143

Maybe add the --no-deps to be on the safe side.

ruben-arts commented 4 weeks ago

@tdejager Then you would need to add some more packages as conda package. But it is indeed a good idea!

jeremyben commented 4 weeks ago

Right, so the only way for now would be to run:
pixi add uv setuptools
and then run the following task, which you could put in a pixi task.
pixi run uv pip install --no-build-isolation git+https://github.com/facebookresearch/detectron2@bcfd464d0c810f0442d91a349c0f6df945467143

Thank you for this workaround, but that defeats the purpose of the lock file and even the dependencies metadata, which is the reason I want to migrate to Pixi (right now I'm pinning my dependencies in environment.yaml).

I'll wait for an option to build without isolation. Thank you for all your work and the security consideration you put into it !

prefix-dev / pixi