Support cross compiling

[isuruf]

Even though meson supports cross compiling, it seems meson-python does not.

https://github.com/mesonbuild/meson-python/blob/78861f5ee4e5257cdebe3ab9faf84027466f1c0b/mesonpy/__init__.py#L313-L348 assumes that the extension is run on the same interpreter. We use crossenv to cross compile in conda-forge and crossenv monkey-patches some things but monkey-patching importlib.machinery.EXTENSION_SUFFIXES does not seem like a good idea.

cc @h-vetinari, @rgommers, @eli-schwartz

[dnicolodi]

Great! It is nice to see that things works as designed also in an environment very different from the one where we usually test meson-python. Thank you for sticking with us and accepting the criticism!

[h-vetinari]

With the release of meson-python 0.13.0, our scipy builds in conda-forge now broke. They were previously relying on the apparently only accidentally-working circumstance that reconfiguration did not error, but in doing so, it allowed us to set up the compilation as needed, and not get tripped up by the meson-python invocation through scipy.

Of course it would be good to avoid the reconfiguration (and we're trying to do that now in the linked PR), but AFAICT, that runs into the issue that meson-python unconditionally adds a --native-file argument that ends up superseding the --cross-file we've provided. This leads to the weird combination that meson sets up build/host/target correctly:

Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: aarch64
Host machine cpu: aarch64
Target machine cpu family: aarch64
Target machine cpu: aarch64

but will then nevertheless start compiling for x86 on aarch/ppc

[10/1628] Compiling C object scipy/_lib/_test_ccallback.cpython-38-x86_64-linux-gnu.so.p/src__test_ccallback.c.o
[11/1628] Linking target scipy/_lib/_test_ccallback.cpython-38-x86_64-linux-gnu.so
[12/1628] Compiling C object scipy/_lib/_fpumode.cpython-38-x86_64-linux-gnu.so.p/_fpumode.c.o
[13/1628] Linking target scipy/_lib/_fpumode.cpython-38-x86_64-linux-gnu.so

It's completely possible that I misdiagnosed what's going on, but if I'm not too far off, it would be nice to have a way to switch off the automatic --native-file addition. From the docs it appears that it's only possible to append to them.

[rgommers]

@h-vetinari I think you diagnosed it correctly. This seems easy to fix by specifying the desired python binary also in a cross file, since Meson will prefer the cross binary over the native binary for a cross build when both are specified:

(scipy-dev) $ cat native-file.txt 
[binaries]
python = '/home/rgommers/mambaforge/envs/scipy-dev/bin/python'

(scipy-dev) $ cat cross-file.txt 
[binaries]
python = '/home/rgommers/mambaforge/envs/cross-env/bin/python'

(scipy-dev) $ meson setup buildc --cross-file=cross-file.txt --native-file=native-file.txt
...
Program python found: YES (/home/rgommers/mambaforge/envs/cross-env/bin/python)

  User defined options
    Cross files : cross-file.txt
    Native files: native-file.txt

Note that cross files "layer", so specifying an extra cross file with only the python binary in it will not interfere with the other cross file that is obtained from the conda-forge infra.

[dnicolodi]

but will then nevertheless start compiling for x86 on aarch/ppc

This is not what I think is happening. Most likely Meson is using the cross compilation toolchain to compile binaries for aarch64 but naming the Python extension modules for the Python interpreter that meson-python specified in the native file.

PEP 517 and related PEP specify that the build needs to happen for the Python interpreter that is executing the Python build backend. meson-python uses the native-file for instructing Meson to compile for the interpreter that is executing meson-python. Meson uses introspects (executing some code with it) the interpreter and determines compilation flags and extension module filenames. meson-python similarly introspects the interpreter and determines the wheel tags. The wheel tags need to agree with the extension modules file names (and of course with the binary objects in these).

Unfortunately, there is no specification for cross-compilation, other than running the build backend with the target Python interpreter (via emulation or something). Any other solution works against the specification and is destined to break from time to time as the build tools evolve.

The solution proposed by @rgommers works (I think, I haven't tested it) for instructing Meson to build for the Python interpreter specified in the cross file. However, meson-python still introspects interpreter that is executing meson-python to generate the wheel tags, thus, if no other trick is applied (there are undocumented environment variables that affect what the introspection return just enough to trick meson-python to do what is expected in cross-compilation, at least on macOS and on Linux), the wheel tags will be off.

If anyone figures out a reliable way to make all this work, I'll be very interested in adding the solution to the docs, and possibly in a test case on our CI infrastructure, so that we can at least be aware of things breaking.

[h-vetinari]

The solution proposed by @rgommers works (I think, I haven't tested it) for instructing Meson to build for the Python interpreter specified in the cross file. However, meson-python still introspects interpreter that is executing meson-python to generate the wheel tags

This might be a silly question (as I don't understand the whole design space here; if so, apologies), but since meson proper has cross-compilation figured out, why not piggyback on that in meson-python?

As in: require that (when meson-python sees a cross-file) that it contain python under [binaries], and then use that to generate the wheel tags (and ignore the native-file bits). That would seem a consistent API design to me.

The absence of standards in this case might be a good thing for once, because that means you're not breaking anything or anyone by building on top of mesons cross-compilation interface. If a python standards ever appears, we could figure out how to transition to that, but that's rather pie in the sky right now, as opposed to the very real need to cross-compile python packages today. :)

[rgommers]

This might be a silly question (as I don't understand the whole design space here; if so, apologies), but since meson proper has cross-compilation figured out, why not piggyback on that in meson-python?

While cross compilation support is robust in general in Meson, unfortunately it isn't yet fully figured out for Python specifically. Before making more changes, I'd really like to see it finalized and documented in Meson itself: https://github.com/mesonbuild/meson/issues/7049#issuecomment-1476022775

[andyfaff]

In https://github.com/andyfaff/scipy/pull/51 I'm trying to cross compile scipy with build:x86_64 and host:aarch64. The idea is to do a two step CI run on cirrus-ci. The first step is to build a wheel on x86_64 then test natively on aarch64 (both Ubuntu).

I can get scipy to complete the build step. I found I had to set _PYTHON_PLATFORM_HOST to get the correct wheel name. However, all the shared libs have x86_64 in the extension suffix, i.e. _test_deprecation_call.cpython-310-x86_64-linux-gnu.so instead of _test_deprecation_call.cpython-310-aarch64-linux-gnu.so (i.e. https://github.com/mesonbuild/meson/issues/7049). When I run file *.so on the shared libs within the unpackaged wheel they are all reported as having aarch64 architecture.

When I install the wheel on linux_aarch64 the tests don't work because all the .so files from the wheel aren't loadable.

I tried providing the python executable in the [binaries] section of the cross and host files, but the meson configure process doesn't like that, probably because the host Python is aarch64 and won't run on x86_64.

meson.build:23:13: ERROR: <PythonExternalProgram 'python' -> ['/root/miniconda3/envs/host-env/lib/python3.10']> is not a valid python or it is missing distutils

I know this cross-compilation process is still being worked on. If you have any tips for how I can proceed, or you would like to use that PR link to try and advance cross-compilation, I'm willing to work on it.

EDIT: you can see a build log here, which also has a build artefact associated with it.

[rgommers]

I tried providing the python executable in the [binaries] section of the cross and host files, but the meson configure process doesn't like that, probably because the host Python is aarch64 and won't run on x86_64.

Yes, this is the exact problem I was running into and reported on in detail in https://github.com/mesonbuild/meson/issues/7049. This does work when using crossenv (as conda-forge does), but not without it. We're kind of stuck for the moment on the non-crossenv case until that issue gets sorted out.

For more context, it is possible to get a non-crossenv cross build to work, but it requires doing something like this (quote from https://github.com/scipy/scipy/issues/16783):

On Void Linux, we set a lot of environment variables to tell the host Python to use the sysconfig data for the build arch and also add the build root to PYTHONPATH, allowing the host Python to find these modules and grabbing relevant information (field sizes, shlib suffixes, etc.) for the build arch rather than the host.