search

conan-io / conan-center-index

Recipes for the ConanCenter repository
https://conan.io/center
MIT License
945 stars 1.71k forks source link

[question] MPI and Virtual Packages #18997

Open nmm0 opened 1 year ago

nmm0 commented 1 year ago

What is your question?

Hi! I wanted to bring up a discussion of MPI and how it can be properly supported in a package manager like Conan. MPI is essential to many C/C++ scientific computing codes, but actually managing the dependency with Conan is tricky.

There is the OpenMPI package (not yet compatible with Conan 2.0), which is convenient for users to build scientific software right out of the box on their local machine.

However, many computer clusters and supercomputers use specially-configured MPI implementations on their systems. Additionally, there are many flavors of MPI available; of course there is OpenMPI, but also mpich, mvapich, a variety of vendor-specific CUDA-aware MPIs, and so on. This means that packages that have a hard dependency on OpenMPI cannot work properly on these systems.

One possibility would be to have an mpi/system package in Conan. In fact, I've developed my own version of this package and it has worked pretty well. However, it wouldn't really be swappable with a specific MPI package such as the OpenMPI package. This makes the "out of the box" install on a development machine without a pre-existing MPI installation a bit more difficult. I also want to note that actually detecting MPI installs is not exactly straightforward due to the large variety of vendor-specific software stacks in HPC environments. In my implementation I actually delegate to CMake (which has about 2000 lines of cmake script dedicated to correctly finding the MPI installation)

Spack, a package manager commonly used in scientific computing and HPC applications, addresses this problem with the idea of "virtual dependencies". These virtual dependencies can be "provided" by a specific package. Would such a scheme be feasible in Conan? This would let users "select" between different MPI libraries or possibly an mpi/system package.

This issue kind of straddles the boundary between the conan repo and conan-center-index, so if it's in the wrong place, please let me know and I will move it to the other repository.

valgur commented 1 year ago

Thanks for raising this. I've had almost the exact same question for a while, but for OpenMP (should there be an openmp/system meta-package, should it also include llvm-openmp).

The ideal solution, if at all possible, would be to leave that choice entirely to the consumer by extending the existing functionality of the provides attribute to swap out compatible packages in the dependency graph. Perhaps something like self.requires("mpi-xyz/x.y.z", replaces=["mpi"]) to replace all packages with provides = ["mpi"]. That would make life easier for both the user by allowing any packages, including in-house ones, to be used and also would save quite a bit of headache for maintainers.

Also, since you might be interested, a review on #18980 would be welcome and appreciated.