Closed carterbox closed 2 years ago
Maybe it rebuilding with updated pinning (from last week) will fix this problem. There was a change the compiler version in the pinnings.
https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/2010
I doubt #92 is the right solution but it wouldn't hurt to consult @conda-forge/core: why would the compiler pinning update require other packages to rebuild? Shouldn't it be a smooth, non disruptive process?
It looks like openmpi-mpifort
is a metapackage which ensures that the fortran compiler is the same one that was used to compile openmpi
. So, if openmpi
was never built with gfortran=10
, then you cannot find an openmpi-mpifort
that is compatible with gfortran=10
.
The problem that I'm experiencing is that a feedstock is using the {{ compiler('fortran') }}
directive which does want gfortran=10
AND openmpi-mpifort
, but there is no openmpi
that was built with gfortran=10
yet.
I think the pinnings determine which compiler versions are used to build feedstocks, right? So if they recently updated gfortran=10
to the pinnings, then we would need to rebuild openmpi
to have a build compatible with gfortran=10
?
Unfortunately, it seems like this feedstock is building only with gfortran=7
, and I don't understand why. Basically, I don't understand what these selectors in the pinnings are doing:
- 10 # [linux]
- 5 # [win64]
- 7 # [os.environ.get("CF_CUDA_ENABLED", "False") == "True" and linux64]
- 9 # [os.environ.get("CF_CUDA_ENABLED", "False") == "True" and linux64]
- 10 # [os.environ.get("CF_CUDA_ENABLED", "False") == "True" and linux64]
- 10 # [os.environ.get("CF_CUDA_ENABLED", "False") == "True" and linux64]
EDIT: These selectors are saying that non-cuda linux builds with gfortran=10
, then each of the cudatoolkit versions has a different compiler version.
IMO, the problem is that openmpi
has no non-cuda build for linux64. It is only being built with cuda=10
. Thus, while the other non-cuda packages are now building with gfortran=10
, openmpi
only builds with gfortran=7
.
So it becomes impossible to have a non-cuda package with openmpi-mpifort
because non-cuda implies gfortran=10
and openmpi-mpifort
implies gfortran=7
.
Thanks, @carterbox. It kinda makes sense to me. Do you know if an MPI compiler wrapper has to pin the host compiler version? Can we add c/c++/fortran compilers to ignore_run_exports_from
? cc: @dalcinl
Otherwise, we'd have to keep rebuilding for every compiler pinning upgrade, which would be annoying...
also cc @conda-forge/core
IMHO, the MPI compiler wrappers should require a host compiler version greater or equal to the one used to build MPI. Does it make sense? Is it doable? I'm not sure how to implement that in the recipe.
Sorry for late reply. Have been unable to access my computer since last week.
IMHO, the MPI compiler wrappers should require a host compiler version greater or equal to the one used to build MPI.
I think it's doable with a little hack. The challenge is that the compiler instruction
run:
- {{ compiler('c') }}
cannot have a version constraint (ex: {{ compiler('c') }} >=9
is not supposed to work).
However, we can first add all compiler instructions (c, cxx, fortran) to ignore_run_export_from
(which is what we do to ignore nvcc), and then, for example for openmpi-mpicc
, add this to the run
section:
run:
# I hard-coded the lower bounds here, which should be inferred from the compiler version at build time
- gcc_linux-64 >=9 # [linux64]
- gcc_linux-aarch64 >=9 # [aarch64]
- gcc_linux-ppc64le >=9 # [ppc64le]
- clang_osx-64 >=11 # [osx64 and not arm64]
- clang_osx-arm64 >=11 # [osx64 and arm64]
and the generated wrapper will have the correct constraint, ex: for linux64 we will get gcc_linux-64 >=9
instead of 9.*
as seen currently:
The same applies to all other compiler wrappers (mpicxx, mpifortran). @carterbox would you like to give it a shot?
run:
- {{ c_compiler }} >={{ c_compiler_version }}
(The lower bounds should be fetch-able using c_compiler_version
, cxx_compiler_version
and fortran_compiler_version
, but I am not sure if we need a Jinja expression or not.)
Always use the jinja2 as above.
That's neat!
I realized that another solution may be
to never do this:
build:
- openmpi-mpifort
- {{ compiler('fortran') }}
but do this instead:
build:
- openmpi-mpifort
I'm not sure if that was the original intention of these compiler meta-packages or not.
No I think the former is preferred. A project could have several shared libraries that do not need to be complied by an MPI compiler wrapper, and it's sufficient to cover those cases with {{ compiler(...) }}
.
Now that I take a second look at it, we probably should have added
build:
- {{ mpi }}-mpicc
to mpi4py-feedstock...😂 @dalcinl
IIRC, the original intention of the {{mpi}}-mpicc
metapackages was to provide a quick way to bring in a compiler within a user's environment to quickly build other MPI-based codes (perhaps via pip, or perhaps with the usual configure & make stanza). The {{mpi}}-mpicc
metapackages should NOT be used as dependencies in recipes. IIRC, all of the other recipes I deal with (mpi4py, petsc[4py], and slepc[4py]) do not use these metapackages as dependencies.
In short, the {{mpi}}-mpicc
should be used by users in quick need of a compiler to install stuff manually within an environment. They should NOT be used by conda recipe authors.
Then how do recipe authors ensure that a compatible compiler is installed with the openmpi? This recipe currently doesn't offer any run constraints for the compilers.
@carterbox I just stated what was the original intention of these metapackages. Since then, things have changed a lot in conda-forge, so much to the point that I no longer feel qualified to provide advice about the current state of affairs.
The root issue here is indeed related to dependency management. You are right, this recipe does not have any run constraint on compilers. In principle, those constraints should not be strict, as long as you use a compiler version newer than the one used to build the package, and the compiler truly commits to backward compatibility. But I do not know how (or whether it is possible) to specify any kind of constrains on compilers (or just their runtimes) within a recipe.
So it looks like until things gets better, the only viable solution to workaround this mess is to rebuild packages for each compiler version as they pop up, which is exactly what you are ~doing in this PR~ proposing in this issue and I never objected.
However, what happens if folks simple do not use {{mpi}}-mpicc/fort
in their recipes? GCC has a high level of backward compatibility, and unless gfortran had recently changed the file format for modules, I guess you can use let say compile new code gcc-11 and use an openmpi package that was built with let say gcc-9 or gcc-10. I may be wrong, but prima facie, I would expect it to work.
Recipe authors should be following these docs. If those somehow are missing things, we should update them.
So maybe I should add something like:
MPI Compilers
-------------
There is no conda-forge equivalent of mpicc/mpif90. Use compiler directives for
coresponding compilers as normal. Do not use the `[openmpi,mpich]-mpicc` metapackages
in the `requirements/build` section of a recipe.
There is no conda-forge equivalent of mpicc/mpif90.
What do you mean? The mpich and openmpi packages do install mpicc/mpicxx/mpifort. I find that first sentence confusing.
MPI Compilers
-------------
Do not use the `[openmpi,mpich]-mpicc` metapackages in the `requirements/build` section
of a recipe; the mpi compiler wrappers are included in the main openmpi/mpich packages.
Add openmpi/mpich to the `requirements/host` section and use compiler directives for the
corresponding compilers in `requirements/build` as normal.
Issue: In https://github.com/conda-forge/staged-recipes/pull/14726, the linux build cannot resolve dependencies in the build environment:
The resolution error is:
Conda forge pinnings current show that fortran compiler version should be 10.
So why is this package still depending on versions 7,9? Is there a reason or just the package needs to be rerendered?
Environment (
conda list
):Details about
conda
and system (conda info
):