Meson integration on windows - discussion about compiler stack

[h-vetinari]

Scipy has switched to meson as of scipy 1.9, and I know from conversation with @rgommers that upstream is eager to drop support for the old distutils builds.

As things stand, conda-forge is not yet able to build scipy using meson, especially on windows, and as such this would put is in a big enough pickle that we'd probably plead with upstream not to drop distutils support yet - I think it's likely that this would be heard, but at the same time, I discussed opening this issue with Ralf so that we can at least plan the necessary steps forward on conda-forge side.

The main issue being that meson relies on having a coherent compiler stack for C/C++/Fortran, and the msvc+gfortran mix that was heroically hacked together by @isuruf (not having had windows-builds for scipy in conda-forge before) is not supported, and will not be.

On windows this is problematic, because we don't have a lot of fortran compiler options, or at least we didn't for a long time. Possible options nowadays:

• MinGW (w/ gfortran)
  • I could imagine that this won't work for ABI reasons with the rest of the MSVC-derived conda-forge ecosystem, though notably, upstream scipy uses a full mingw stack.
  • Upsides: ?
  • Downsides: might be ABI-incompatible
• MSVC (or LLVM?) + flang
  • fortran support in LLVM has been a long time coming; so long, that some people have lost faith in it. However, things are progressing, and now there's experimental code generation support in LLVM 15. Indicatively, Fujitsu recently let flang loose on one of their test suites and found that 3447/8247 programs failed. There's still a lot of work to do, but it might be usable, especially because the fortran in scipy is rather old and doesn't use features from newer fortran standards. In particular, raising bugs in LLVM would probably make it possible to sort most things out by LLVM 16.
  • Upsides: LLVM is ABI-compatible with MSVC, license-compatible with redistribution & we use llvm-openmp on windows in various places already
  • Downsides: LLVM 15 (and especially flang) hasn't been built in conda-forge yet (infra changes, lack of time on my side), maturity of compiler might be unsatisfactory
• MSVC (or LLVM?) + lfortran
  • LFortran aims to be a general-purpose Fortran compiler, and is particularly putting emphasis on supporting the compilation of scipy.
  • Upsides: it's in conda-forge already and actively maintained (CC @conda-forge/lfortran)
  • Downsides: support for compiling scipy is unclear, and I'm assuming meson would need to be taught to recognise LFortran (at least, it doesn't really show up in the repo; CC @eli-schwartz), ABI impact is unclear to me (but probably fine since LLVM-based), maturity of compiler might be unsatisfactory
• MSVC + ifort
  • Ralf pointed out to me that Intel's ifort is now much more available since it moved under the oneAPI umbrella. It's licensing situation doesn't allow redistribution of the compiler though, so AFAICT we'd be in a similar situation as with nvcc - meaning we'd have to do this in the ci setup (downloading binaries). Not sure if there have been thoughts about packaging ifort for conda-forge, similar to what's happening for MKL (CC @conda-forge/intel_repack).
  • Upsides: mature compiler, interaction with MSVC/MKL works
  • Downsides: not easy to integrate with conda-forge, might need checking with Intel re:license in any case

Other issues:

• [ ] cross-compiling osx-arm64, linux-aarch64, linux-ppc64le
• [ ] support for alternative BLAS/LAPACK implementations in meson
• [ ] Being able to match MKL's Fortran ABI, see e.g. here and here

CC @conda-forge/core

[eli-schwartz]

There is a draft PR to add LFortran to the list of supported compilers in Meson; you can find it here: https://github.com/mesonbuild/meson/pull/9137

It hasn't been merged yet because the author said "This needs a custom linker before it is ready."

[beckermr]

Are there any viable binary repackaging options that could tide us over? Please excuse my ignorance if this is a dumb question. :)

[h-vetinari]

Are there any viable binary repackaging options that could tide us over?

Unfortunately, I don't think so. Scipy ships with openblas, and I don't think we can keep our blas flavours (e.g. default to mkl on windows) without rebuilding ourselves. There'd probably be other issues as well (not necessarily matching our other pinnings, esp. numpy).

[pkgw]

That Meson PR looks pretty stalled, though hopefully @eli-schwartz's ping of the original author will prove that wrong. But there's also https://github.com/lfortran/lfortran/issues/704 which seems to indicate that maybe there's some more recent interest coming from the lfortran team.

[certik]

Do you know what needs to be done? I am focusing on compiling Minpack (from SciPy): https://github.com/lfortran/lfortran/issues/512, after that we'll make sure it runs everywhere correctly. If you want to help in the meantime, it should not be difficult to get LFortran working with Meson.

[h-vetinari]

Thanks for chiming in @certik!

Do you know what needs to be done?

That's a pretty broad question (so I hope I'm interpreting it correctly), but assuming lfortran is chosen (i.e. at minimum if it turns out there are no ABI-issues, e.g. with MKL), it would - very broadly - be:

1. Get meson to recognise lfortran (help appreciated, no idea what's necessary to do it)
2. Get lfortran to compile SciPy without errors (also not something I can offer to do realistically, except perhaps bug reports)
3. Update the feedstock here (this I can do provided 1. is in place and then help iterating on 2.)

[certik]

Thanks @h-vetinari. Yes, I meant with just 1. I have to do 2., we are working on it.

[rgommers]

Get meson to recognise lfortran (help appreciated, no idea what's necessary to do it)

This shouldn't be too hard, let's not worry about this part.

On windows this is problematic, because we don't have a lot of fortran compiler options, or at least we didn't for a long time. Possible options nowadays:

I think there's a clear distinction between short-term and long-term options here. LFortran seems to have very nice momentum, and also is built by folks who care about SciPy and the wider Python ecosystem. It still seems like a good amount of work to get all of SciPy to work - but long-term it seems like the most promising option.

Short-term options:

• Mingw-w64: SciPy uses it to build wheels, and that works (at least, nothing exploded so far). So that seems like a reasonable option, if a suitable version of it is packaged in conda-forge and there are no showstoppers.
  • Main hiccup I would expect: SciPy uses a custom OpenBLAS build for its wheel, and it's unclear to me if we can get away without that.
• Intel Fortran (classic): downloading in CI doesn't seem like a huge issue to me. Meson does this in its own CI, and I'm also trying to figure it out for SciPy: https://github.com/scipy/scipy/pull/16957. Not great long-term to rely on a proprietary compiler only, but as a stopgap, why not?
• Flang: if it works then great, but honestly I've stopped paying attention.

[h-vetinari]

Just saw that intel fortran is now available in conda-forge: https://github.com/conda-forge/intel-compiler-repack-feedstock

That might accelerate things a bit, but will still take a bit of time (I don't think in time for 1.11, but hopefully soon after).

[rgommers]

I don't see that it's now the actual Intel compilers that are available, rather than just the runtimes. The package description claims the former, but intel-compiler-rt suggests the latter. I'm on macOS arm64 right now, so can't install the package to check. Can someone confirm?

If we do have Intel compilers now, I'd say we should build SciPy with MSVC + Intel Fortran on Windows. SciPy does not depend on OpenMP, so we should make sure it doesn't drag in libiomp (that tends to get problematic in combination with other packages that pull in libgomp). Everything else should work I'd think.

[h-vetinari]

I don't see that it's now the actual Intel compilers that are available, rather than just the runtimes.

I don't know that feedstock yet, but despite the name intel-fortran-rt, the description is:

summary: Runtime for Intel® Fortran Compiler Classic and Intel® Fortran Compiler (Beta)

[h-vetinari]

Ah, I just realized that I assumed an Oxford Comma here ("runtime for x, and Intel Fortran Compiler"). Anyway.

I can try to find out how that runtime is supposed to be used (presumably it only makes sense to get this into conda-forge if the compiler can be used as well...?).

[rgommers]

I can try to find out how that runtime is supposed to be used (presumably it only makes sense to get this into conda-forge if the compiler can be used as well...?).

That'd be useful I think. Maybe that info could end up in the feedstock README or the conda-forge docs?