Add "external" dummy builds

[kburns]

Comment:

This feedstock has been tremendously valuable for building stacks for Dedalus, but in some cases when building on HPC systems, it may be preferable to use an existing HDF5 build from the cluster. However, this currently makes it hard to use other conda builds that rely on HDF5, since conda tries to install it's own HDF5 binaries.

I think this could be avoided by creating "external" dummy builds on this feedstock like what has been done with MPI. Then upstream conda builds that rely on HDF5 should work with either the true builds from the feedstock, or just pass through to the existing system libraries if the user pre-installs the dummy build.

I'm still pretty new to conda forge builds, so I might eventually be able to try this myself and open a PR, but if it's trivial for anyone else to do in the meantime, it's a feature that I think would be greatly appreciated!

[hmaarrfk]

I am very sympathetic to this, but i don't think you need help from conda forge for this to immediately help your users.

In general, for specialized system, i would highly suggest creating your own conda package that resolves the problem.

It your users add your channel, then it will remain compatible. It's just impossible for us to test "compatible with external stuff" especially when API/ABI and feature set compatible is concerned.

[kburns]

Whoops sorry the link was broken before, here's the right link to what conda-forge MPIs are doing for these external builds.

If I understand it correctly, the builds really are just empty, and it's fully the responsibility of the user at that point to install a dummy build thats ABI compatible with their system packages. I think this could be quite broadly useful for getting better builds using conda-forge in large HPC environments, since HDF5 is a low-level piece of many scientific software stacks.

[hmaarrfk]

Thank you for the updated link. I seem to have forgotten that.

since HDF5 is a low-level piece of many scientific software stacks.

I'm just not sure where to draw the line. The advantage of open source software is that it encourages upstreaming changes. One question to ask is are you not able to submit patches to HDF5? I would much rather a solution where you've found optimizations that could be shared beyond the immediate users of your HPC.

I'm also not sure why the "dummy package" needs to exist in the conda-forge channel. HPCs often have installation guidelines. I think that this kind of documentation should be available in your HPC installation instructions.

For conda, we encourage users to utilize the conda-forge channel with our personal channel enabled. Our personal channel provides compatible packages that are optimized for use with our HPCs. For an optimized version of hdf5 make sure to use the constraint:

conda config --add channels conda-forge
conda config --add channel mithpc_optimized
conda config --set channel_priority strict 

Then in your environment

mamba install hdf5=*=mithpc*

I might be the wrong personal to ask to sympathize with HPC systems. My experience with them (10 years ago) was that it seemed like the users were not in charge and the HPC staff was actively working against you installing your own libraries. I've also been working to enable optimizations in the larger open source community when I find HDF5 optimizations (e.g. enabling the HDF5 DIRECT driver on conda-forge and building in shims for it in h5py and netcdf4). I know it takes time to get them in, but again there is a way for you to provide your own packages to your own users today without using conda-forge.

I think @beckermr might be in favor of your proposal given that he made the PR for mpich.

[beckermr]

MPI on HPC systems tends to integrate tightly with the underlying software that provisions jobs, gets their data to the nodes, starts the jobs, etc. This is why the shims were installed. Literally, people cannot get jobs to run without using the shims on essentially every big HPC system I have ever used. I consider this a last resort as opposed to a generalized solution.

Enough folks (literally anyone doing science on an HPC cluster) hit this problem that it made sense to ship a solution in conda-forge.

I am not opposed to doing this for HDF5 necessarily but I would like to understand why this is needed precisely. If a machine isn't working at all with conda, that is one thing. If the build is simply suboptimal, then I think we are in a different regime.

[beckermr]

One way to test things would be to setup your env, forcibly uninstall the relevant hdf5 packages, ensure the system ones are in your linking path, and then run jobs comparing them to the conda libraries. This will give you a sense of the change in performance or w/e it is that is at issue.

[kburns]

Ah sorry, to clarify, I am not an HPC admin and haven't created any of these specialized MPI or HDF5 builds. I just develop PDE solvers that use MPI and HDF5, and we've experienced that many people using our solvers are running on various different HPC systems that have their own specialized MPI implementations.

Conda-forge's support of external MPI passthroughs really helps there, but right now it also just kicks the can 1 step down the road to HDF5, which also needs to be linked to these special MPI implementations to use MPIO. Since I'm not trying to fix this with builds for any specific system, I'm not sure making new packages in a new channel would work.

I'm just not sure where to draw the line.

Point very well taken! At least in our case, and I think for quite a few similar packages, there's just a couple low-level libraries that need to be built against the cluster-optimized MPI, basically just FFTW and HDF5. Other folks might need more. But just like the MPI variant selection that several packages are now using quite effectively within conda-forge, maybe adding these dummy builds won't require too much work but enable a lot more flexibility for HPC.

[beckermr]

Can you clarify how HDF5 needs to be built and/or run in order to use the specialized MPIO features? Does it have to be compiled against the system MPI libraries directly or does it only need to link against the system MPI libraries?

[kburns]

I think the problem is when the HPC system has an MPI that's not ABI compatible with mpich or openmpi, so we can't just use conda-forge's parallel HDF5 builds with one of those dummy MPI builds. Instead I think we would need a dummy HDF5 build so we can use the system HDF5 along with the system MPI, and still be able to satisfy the HDF5 requirement of upstream conda packages (that conform to the corresponding HDF5 ABI). Does that sound right?

[hmaarrfk]

forcibly uninstall the relevant hdf5 packages, ensure the system ones are in your linking path,

conda activate my_environment
conda uninstall hdf5 --force --offline --yes

should do that.

[beckermr]

This doesn't make sense. If the HPC MPI is not ABI compatible with the conda-forge one then you can't use any conda forge software with it, HDF5 or no HDF5.

If you have a specific error on a specific system, then send it along so we can discuss further.

[kburns]

Ok it might not be much of an issue since it looks like recent versions of MVAPICH, MPT, and Intel MPI are ABI-compatible with MPICH, which I didn't realize. I'll check with some of the users on these systems and see if the current MPICH dummy builds + HDF5 work.