search

rapidsai / build-planning

Tracking for RAPIDS-wide build tasks
https://github.com/rapidsai
0 stars 4 forks source link

Implement a cuda-version wheel metapackage #28

Open vyasr opened 8 months ago

vyasr commented 8 months ago

Currently we have a cuda-version metapackage in conda that can be used in install commands to select the appropriate version of other packages. Ideally such a package would facilitate something like the following:

pip install cudf cuml cuda-version=12.2

with the appropriate cudf/cuml packages being pulled for us. The major caveat with pip vs conda is that we do not have a single package across CUDA versions for a given RAPIDS package, but instead separate packages by major version. That is because wheels do not offer any way of encoding arbitrary extra tag information to distinguish wheels that are identical according to the standard information that wheels support (platform, Python version).

In lieu of the latter being feasible, we would at minimum like to support a package that would serve to constrain our environment and prevent inconsistent installations, e.g.

pip install cudf-cu12
pip install cuml-cu11  # We would want this to fail because we have cudf-cu12

That is a more tractable problem for which the solution should be achieved by adding the appropriate cuda-version constraint to each *-cu* package.

msarahan commented 8 months ago

possibly relevant discussion: https://discuss.python.org/t/dependency-constraints-for-setuptools/21897

msarahan commented 8 months ago

My initial thought on this topic is that users should not be directly specifying the -cuXY part themselves at all. There should be a detection tool that fills that in for users, such that they only ever specify something like cudf. I'm working on one such tool that's presently internal at NVIDIA. The tool could potentially get out of alignment with the environment if the user updated CUDA version, though, so that's not a slam-dunk. Perhaps some environment recording/inspection that strips any -cuXY modifications when considering specs to solve for, then re-adds them as needed. That's not at all how pip works, though. That kind of idea is more declarative - you tell the tool what you want in the environment, and it figures out how to meet that.

I'm really not sure there is a good way to do this without more general metadata support, but I'm glad people are thinking about it.

vyasr commented 8 months ago

Yeah I'm not sure that there's a great solution to this. I agree that it would be ideal if we could avoid the -cu* suffixes altogether, but short of upstream work in pip itself I don't see a way to do that.

jakirkham commented 7 months ago

This also looks relevant

https://discuss.python.org/t/idea-selector-packages/4463