conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
7.95k stars 951 forks source link

[question] How to make CUDA version affect package_id #16475

Closed aroidzap closed 2 weeks ago

aroidzap commented 2 weeks ago

What is your question?

Hello, I'm trying to find out, how to work with Conan packages, that are built with different versions of CUDA.

Some packages have with_cuda option, but does not specify or check for exact version. This way the package can be built with CUDA 12 and than be consumed on other system with CUDA 11, as there is no way to to propagate the CUDA version to the package_id.

It would be best, if all recipes, that needs CUDA, requires some CUDA meta package. This package could be "provided" using [platform_requires] feature. This would mean that CUDA version would affect the package_id.

This solution is nice, but requires that all packages using CUDA are updated in conan-center-index with this new meta-package require. As I don't want to maintain modified recipes of packages I need, I'm looking for some workaround, that would either:

Unfortunately I did not find any way how to do either of these workarounds. If You have any other idea, how to do this, please let me know, thanks!

Also in some other issues in conan-center-index I found that CUDA support in Conan was promised in the early 2024 Q2, but I don't see any updates on that.

Have you read the CONTRIBUTING guide?

memsharded commented 2 weeks ago

Hi @aroidzap

Thanks for your question.

Hello, I'm trying to find out, how to work with Conan packages, that are built with different versions of CUDA.

I think that it depends on the use case.

If the packages are very specific to be built with CUDA, because they have CUDA code in it, then maybe the best would be to have a cuda custom setting in settings.yml and then it can be added directly in the recipes like settings = "os", "compiler", ..., "cuda" or something in that line

Then, there are packages that are not CUDA-accelerated, for example, a normal zlib recipe will not be affected by CUDA at all, even if some other packages in the dependency graph. So we don't want to inject the CUDA version to absolutely all packages in all situations. It seems that it should be packages that define that they want to be affected by CUDA.

There are several mechanisms to affect the package-id. Recipes can for example define what they want in their package_id() method, and force an option value, or even a conf to be part of the package_id. So the value of the CUDA version could be provided for example via a user.xxx:yyy conf value.

It would be best, if all recipes, that needs CUDA, requires some CUDA meta package. This package could be "provided" using [platform_requires] feature

This is not possible, the platform_requires are a replacement for some already existing requires in the recipes, but the recipes should have already declared some requires in them.

@jcar87 has been doing some progress in some "foundations" of using CUDA with Conan, but nothing yet for ConanCenter.

aroidzap commented 2 weeks ago

Thanks for a quick response. In my use-case I want to avoid incompatible packages being used. I don't mind that much if there are redundant / duplicate packages that are same but have different package_id.

For me the best workaround would be to add something to the profile, that would affect all package_ids (maybe add some suffix) without the need of modifying any package recipe.

This way I could add CUDA version, CUDA architectures or whatever and packages won't be consumed if there is some possible incompatibility.

Maybe even better would be some feature so I can globally inject some require to all packages being used. Maybe some kind of pre_require hook https://docs.conan.io/2/reference/extensions/hooks.html?

memsharded commented 2 weeks ago

For me the best workaround would be to add something to the profile, that would affect all package_ids (maybe add some suffix) without the need of modifying any package recipe.

That is straightforward:

That will create all packages with that specific package_id for that specific cuda version. This will be a bit inefficient, having to build more binaries than necessary for many libraries that don't depend on CUDA, but if it is what you want, it should take you only a couple of minutes to add it. You can check more on https://docs.conan.io/2/reference/binary_model/extending.html#custom-settings

aroidzap commented 2 weeks ago

Thank you very much!

memsharded commented 2 weeks ago

Happy to help!

Let us know if it works as expected, don't hesitate to create new tickets for any further question or issue you might have.