conda-forge / scipy-feedstock

A conda-smithy repository for scipy.
BSD 3-Clause "New" or "Revised" License
6 stars 36 forks source link

Meson integration on windows - discussion about compiler stack #213

Closed h-vetinari closed 1 year ago

h-vetinari commented 2 years ago

Scipy has switched to meson as of scipy 1.9, and I know from conversation with @rgommers that upstream is eager to drop support for the old distutils builds.

As things stand, conda-forge is not yet able to build scipy using meson, especially on windows, and as such this would put is in a big enough pickle that we'd probably plead with upstream not to drop distutils support yet - I think it's likely that this would be heard, but at the same time, I discussed opening this issue with Ralf so that we can at least plan the necessary steps forward on conda-forge side.

The main issue being that meson relies on having a coherent compiler stack for C/C++/Fortran, and the msvc+gfortran mix that was heroically hacked together by @isuruf (not having had windows-builds for scipy in conda-forge before) is not supported, and will not be.

On windows this is problematic, because we don't have a lot of fortran compiler options, or at least we didn't for a long time. Possible options nowadays:

Other issues:

CC @conda-forge/core

eli-schwartz commented 2 years ago

There is a draft PR to add LFortran to the list of supported compilers in Meson; you can find it here: https://github.com/mesonbuild/meson/pull/9137

It hasn't been merged yet because the author said "This needs a custom linker before it is ready."

beckermr commented 2 years ago

Are there any viable binary repackaging options that could tide us over? Please excuse my ignorance if this is a dumb question. :)

h-vetinari commented 2 years ago

Are there any viable binary repackaging options that could tide us over?

Unfortunately, I don't think so. Scipy ships with openblas, and I don't think we can keep our blas flavours (e.g. default to mkl on windows) without rebuilding ourselves. There'd probably be other issues as well (not necessarily matching our other pinnings, esp. numpy).

pkgw commented 2 years ago

That Meson PR looks pretty stalled, though hopefully @eli-schwartz's ping of the original author will prove that wrong. But there's also https://github.com/lfortran/lfortran/issues/704 which seems to indicate that maybe there's some more recent interest coming from the lfortran team.

certik commented 2 years ago

Do you know what needs to be done? I am focusing on compiling Minpack (from SciPy): https://github.com/lfortran/lfortran/issues/512, after that we'll make sure it runs everywhere correctly. If you want to help in the meantime, it should not be difficult to get LFortran working with Meson.

h-vetinari commented 2 years ago

Thanks for chiming in @certik!

Do you know what needs to be done?

That's a pretty broad question (so I hope I'm interpreting it correctly), but assuming lfortran is chosen (i.e. at minimum if it turns out there are no ABI-issues, e.g. with MKL), it would - very broadly - be:

  1. Get meson to recognise lfortran (help appreciated, no idea what's necessary to do it)
  2. Get lfortran to compile SciPy without errors (also not something I can offer to do realistically, except perhaps bug reports)
  3. Update the feedstock here (this I can do provided 1. is in place and then help iterating on 2.)
certik commented 2 years ago

Thanks @h-vetinari. Yes, I meant with just 1. I have to do 2., we are working on it.

rgommers commented 2 years ago

Get meson to recognise lfortran (help appreciated, no idea what's necessary to do it)

This shouldn't be too hard, let's not worry about this part.

On windows this is problematic, because we don't have a lot of fortran compiler options, or at least we didn't for a long time. Possible options nowadays:

I think there's a clear distinction between short-term and long-term options here. LFortran seems to have very nice momentum, and also is built by folks who care about SciPy and the wider Python ecosystem. It still seems like a good amount of work to get all of SciPy to work - but long-term it seems like the most promising option.

Short-term options:

h-vetinari commented 1 year ago

Just saw that intel fortran is now available in conda-forge: https://github.com/conda-forge/intel-compiler-repack-feedstock

That might accelerate things a bit, but will still take a bit of time (I don't think in time for 1.11, but hopefully soon after).

rgommers commented 1 year ago

I don't see that it's now the actual Intel compilers that are available, rather than just the runtimes. The package description claims the former, but intel-compiler-rt suggests the latter. I'm on macOS arm64 right now, so can't install the package to check. Can someone confirm?

If we do have Intel compilers now, I'd say we should build SciPy with MSVC + Intel Fortran on Windows. SciPy does not depend on OpenMP, so we should make sure it doesn't drag in libiomp (that tends to get problematic in combination with other packages that pull in libgomp). Everything else should work I'd think.

h-vetinari commented 1 year ago

I don't see that it's now the actual Intel compilers that are available, rather than just the runtimes.

I don't know that feedstock yet, but despite the name intel-fortran-rt, the description is:

summary: Runtime for Intel® Fortran Compiler Classic and Intel® Fortran Compiler (Beta)
h-vetinari commented 1 year ago

Ah, I just realized that I assumed an Oxford Comma here ("runtime for x, and Intel Fortran Compiler"). Anyway.

I can try to find out how that runtime is supposed to be used (presumably it only makes sense to get this into conda-forge if the compiler can be used as well...?).

rgommers commented 1 year ago

I can try to find out how that runtime is supposed to be used (presumably it only makes sense to get this into conda-forge if the compiler can be used as well...?).

That'd be useful I think. Maybe that info could end up in the feedstock README or the conda-forge docs?