Support cross compiling

[isuruf]

Even though meson supports cross compiling, it seems meson-python does not.

https://github.com/mesonbuild/meson-python/blob/78861f5ee4e5257cdebe3ab9faf84027466f1c0b/mesonpy/__init__.py#L313-L348 assumes that the extension is run on the same interpreter. We use crossenv to cross compile in conda-forge and crossenv monkey-patches some things but monkey-patching importlib.machinery.EXTENSION_SUFFIXES does not seem like a good idea.

cc @h-vetinari, @rgommers, @eli-schwartz

[rgommers]

Thanks for identifying that issue @isuruf. It looks like we need to avoid this extension check, and also add a basic cross-compile CI job. I was just looking at that for SciPy. I'm not that familiar with crossenv, but I think that for meson-python CI we need a cross-compilation toolchain (maybe from dockcross) and also both host and build Python already installed - crossenv doesn't do that for you if I understood correctly. So we need a Docker image with those things - perhaps from conda-forge?

[dnicolodi]

I don't think calling this a bug is fair. Cross compilation is not supported by the Python packaging ecosystem. And apparently "crossenv monkey patches some things" to make it work, whatever this means, given that PEP 517 build are usually done out-of-process. The detection of the stable ABI is just one of the places where we introspect the running environment, and Meson does too. Python does not offer a supported way to obtain the information required to build an extension module for another interpreter, let alone for an interpreter running on another platform. If you need support for cross compilation, please bring the issue to the attention of the CPython developers.

[rgommers]

Fair enough, Python's support is very poor. But right now moving from setuptools to meson-python is a regression, because crossenv does this ad-hoc patching of just the things that setuptools needs. So it is important and we need to fix it. Ideally both with and without crossenv.

We're not that far off, I'm just messing around with trying to get a Linux x86-64 to aarch64 cross compile to work for SciPy with:

dockcross-linux-arm64 bash -c "python -m pip install . --config-settings=setup-args=--cross-file=/work/cross_aarch64.ini"

I suspect it only requires a few tweaks.

[dnicolodi]

The first obvious issue I can think about is that we use sys.maxsize to determine if the target Python interpreter is 32 or 64 bits. How do you plan to make that work? AFAIK setuptools uses the same check, thus setuptools does not support cross compiling between 32 and 64 bit architectures.

[eli-schwartz]

The detection of the stable ABI is just one of the places where we introspect the running environment, and Meson does too.

The difference is that meson specifically executes an external python installation with a json dumper script, in order to scrape for information which meson knows it may need. It looks like meson-python checks for this information in-process instead, which means that it is bound to the same version of python that it is running inside.

Cross compilation has two challenges:

• compile for something other than what you are
• compile for something you cannot exec on your hardware/software stack

[dnicolodi]

Well, meson-python implements PEP 517, which does not have any provision for cross-compiling. It assumes that the Python interpreter used for the build, is the one you are building for. More generally, there is no way to generate wheel tags for an interpreter that you cannot execute. And if you can execute the Python you are building for, why not use it to run meson-python? I know all this is not ideal. But supporting these interfaces with these constraints is what meson-python is set up to do. If we want to build a tool for cross compiling Python wheels, it would have to look very different.

[eli-schwartz]

Sure, I do understand and empathize with those issues. There is finally some interest in getting this to work once and for all, though, I think. :)

And if you can execute the Python you are building for, why not use it to run meson-python?

FWIW this is actually a complex topic. Particularly worthwhile to note:

• qemu makes it possible to more or less fully generically emulate other machines, with the restriction that this is Linux-specific
• WINE on Linux can emulate Windows

But actually doing so is slow. So you actually don't want to do this, at least not more than you can get away with. So, there's a benefit to emulating just the json dumper and then running the rest of the build natively.

[dnicolodi]

So, there's a benefit to emulating just the json dumper and then running the rest of the build natively.

I completely agree, but while it makes sense for Meson to work this way, I think it would be overkill for meson-python. But, because PEP 517, we don't even have to think about it: the interfaces we need to implement do not support this.

[rgommers]

There's several levels of making things work here:

1. Make it work with crossenv, with the same level of support as setuptools has
2. Make it work by figuring out what we actually need to know (wheel tags, `sys.maxsize & co) and then allowing a user to put that into a cross file
3. Make it work out of the box without the user having to do (2) (this one requires stdlib support)

(1) and (2) should be feasible in a shorter time frame. (3) is going to be a lot more painful.

[h-vetinari]

If you need support for cross compilation, please bring the issue to the attention of the CPython developers.

By necessity, conda-forge has built a lot of its packaging around cross-compilation (i.e. there aren't native osx-arm64 CI agents, so we need to cross-compile from osx-64). These builds might even flow back to the non-conda world occasionally.

So it's a classic case of Hyrum's law, where a lot of things have grown up around an implicit interface (in this case the possibility to monkey-patch in cross-compilation), that we cannot switch the conda-forge builds for scipy to meson unless we solve the cross-compilation part somehow.

I don't mind carrying patches or workarounds for a while (i.e. it doesn't need to have a sparkling UX), but it would be very beneficial to have it be possible at all.

[dnicolodi]

Cross compiling for an arm64 target on a x86 build machine on macOS is already supported with the same interface used by setuptools.

For other user cases, I'm afraid that making the monkey-patches required to make cross compilation work with setuptools also work for meson-python is not possible, unless someone defines somewhere the interfaces that we can use and the ones we cannot. Even then, the thing would be extremely fragile without a clear definition of how the interface that we can use work when cross-compiling.

importlib.machinery.EXTENSION_SUFFIXES here is only as a safety check. We can remove it. But I would be very surprised if there are no other things that break.

[rgommers]

Even then, the thing would be extremely fragile without a clear definition of how the interface that we can use work when cross-compiling.

This is true, but the good thing is that the build envs are much better under control, and regressions are not as painful. We're looking to support distros here, not make pip install mypkg --cross from source work for end users.

[eli-schwartz]

It saddens me a bit that people seem to think crossenv is the definition of how to cross compile. There are people who have been cross compiling python modules without knowing that crossenv exists (or I think in one case, being vehemently convinced that crossenv is horrible and the worst thing possible for the cross compilation community 🤷‍♂️).

I think the reality is some combination of "a number of different groups have independently discovered some key aspects, and have different ideas how to do the rest".

• There's no such thing as a crossenv interface
• nor a cross-compile setuptools interface regardless of framework used to run setuptools

Meson has a cross-compile interface. Meson defines how to cross compile a meson project.

Frameworks used for cross compiling, including but not limited to crossenv, yocto, buildroot, voidlinux, etc, are responsible for interacting with the Meson cross-compile interface, and that is all. Meson, in turn, considers its cross compile interface to be "run a python interpreter and dump a specific list of values" -- this isn't well documented in the manual, but it does exist.

(Those projects may have also homebrewed their own cross compile interface for setuptools, but that really doesn't matter for either meson-python or for meson. At the end of the day, the tricks they use are irrelevant to meson except for the sysconfig tweaking, and that's just parallel evolution, not something owned by a specific tool.)

IMHO meson-python shouldn't fail to package a project that meson has successfully cross compiled, and for that reason I'm happy to see the check being removed. :)

If you want to validate that the ext suffix matches the binary architecture, that seems like a job for auditwheel or something.

If you need to generate a wheel tag, I actually feel a bit like that doesn't much matter. If you're building for local use you just immediately extract the results and discard the wheel tag in the process (and I assume conda has the same basic rationale to not-care) and again, I thought this was what auditwheel is for. If it can fix the manylinux version, I'd assume it can also fix a broken CPU architecture... can we just fall back on a nonsense wheel tag like "unknown"? Or lie and call it "none-any"?

[rgommers]

It saddens me a bit that people seem to think crossenv is the definition of how to cross compile.

I don't think that. crossenv is indeed just one of the ways, and seems to be a pragmatic hack to work around Python lack of support.

If it can fix the manylinux version, I'd assume it can also fix a broken CPU architecture... can we just fall back on a nonsense wheel tag like "unknown"? Or lie and call it "none-any"?

I agree with most of what you wrote, but not this. auditwheel is specific to manylinux-compliant wheels, and manylinux is not appropriate or needed in a number of circumstances. We do need to generate a correct wheel tag. It shouldn't be that hard.

It seems to me like meson-python does need to know that we're cross compiling. Detecting whether --cross-file is present in config_settings is probably reliable (covers everything except for the macOS environment variable way)? If so, could we just read the host machine section of the cross file, and generate the wheel tag from that?

[dnicolodi]

@eli-schwartz I agree with you on all points, except one: the tools that takes a Meson build directory and packs it up in a wheel is not meson-python, but something else, that may or may not share some code and be implemented in the same project as the current meson-python.

meson-python define itself has an implementation of PEP 517 https://peps.python.org/pep-0517/ In the interfaces defined in PEP 517 there is nothing that allows cross-compilation: it implicitly assumes that the wheels are being built for the Python interpreter running the build process. This is one of the reasons why solutions for cross compiling wheels have the taste of hacks: they need to work-around this interface limitation. AFAICT, auditwheel has the same core assumption, thus I don't think it can fix wheels built for an architecture that is not the one where it is being run.

Building on @eli-schwartz consideration that the correct cross-compilation interface is the one provided by Meson, we need a tools that allows access to that interface (forgetting PEP 517). However, wrapping Meson in another tool dedicated to build Python wheels is not necessary, what the tool needs is just the Meson build directory. I'm thinking to something that could be run like this:

meson setup $build_dir --cross-file $crossbuild_definition
meson compile -C $build_dir
meson-wheel-it-up $build_dir --tag $wheel_tag

Where meson-wheel-it-up is just an implementation of meson install that packs the build artifacts into the right format.

[dnicolodi]

Detecting whether --cross-file is present in config_settings is probably reliable (covers everything except for the macOS environment variable way)? If so, could we just read the host machine section of the cross file, and generate the wheel tag from that?

This would require determining which architecture we are building for from the compiler executable paths. I'm sure it can be done, but the user knows for which architecture they are building, they can just tell the wheel packaging tool about it. Also, it would require determining which flavor of python interpreter (cpython/pypy/pyston,... and the relative version) we are building for from the executable path. Also this seems an information that the user is in a better position to tell us.

More problematic is the handling of build dependencies and (optional) build isolation implemented by PEP 517 frontends. You most likely do not want that for cross compiling. Build dependencies for cross compilation need to be correctly handled considering which dependencies are to be run on the host (cython, pythran) and which are libraries for the target (numpy, scipy, ...). PEP 517 frontends cannot do that, and they mostly get in the way.

[rgommers]

the tools that takes a Meson build directory and packs it up in a wheel is not meson-python, but something else, that may or may not share some code and be implemented in the same project as the current meson-python.

This is a little confusing to me. I'm not sure what you have in mind here exactly. Whatever other project it is, I think that is an implementation detail under meson-python. From my perspective only have two things: Meson as the build system, and meson-python as the layer between pip & co to build sdists and wheels. And the goal of meson-python is to make use of Meson by Python projects as seamless as possible.

What is and isn't in PEP 517 isn't that relevant, there's --config-settings as an explicit escape hatch for anything else that's not directly supported by a standard. And that includes cross-compiling. In fact, proper support for cross-compiling is one of the major benefits of moving from distutils & co to Meson. We certainly get a lot of bug reports and questions about it for NumPy and SciPy, and in the past (pre Meson) I've always answered "don't know, distutils doesn't support that, good luck and if you learn something please share". I now want proper support for it.

AFAICT, auditwheel has the same core assumption, thus I don't think it can fix wheels built for an architecture that is not the one where it is being run.

auditwheel was brought up by both of you, but it really isn't relevant. Its job is specifically to deal with distributing wheels on PyPI. There are many packaging systems other than PyPI, so we cannot rely on auditwheel for anything.

that the correct cross-compilation interface is the one provided by Meson, we need a tools that allows access to that interface

It's not really an interface in the sense that meson-python can use it - but I don't think there's a need for that. We only need a few fragments of information. Basically metadata for platform, OS, and interpreter flavor and ABI - I'm not sure that there's much beyond that. So we should just figure that out from the info config_settings incoming data.

This would require determining which architecture we are building for from the compiler executable paths.

Not really - there's a section like this in the cross file:

[host_machine]
system = 'windows'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'

it would require determining which flavor of python interpreter (cpython/pypy/pyston,... and the relative version) we are building for from the executable path. Also this seems an information that the user is in a better position to tell us.

That's a good point. I think the requirement here is "build interpreter kind == host interpreter kind" for now (in practice, the main demand is CPython). Possibly there's a need to add the host interpreter to the cross file - let's cross that bridge whne we get to it.

More problematic is the handling of build dependencies and (optional) build isolation implemented by PEP 517 frontends.

No one is going to use build isolation with cross compilation I think. It'll be something like:

python -m build --wheel --no-isolation --skip-dependency-check -C--cross-file=cross_aarch64_linux.ini

There's nothing for us to do here to deal with build isolation AFAIK.

[dnicolodi]

I think the requirement here is "build interpreter == host interpreter" for now

The problem raised in this issue is about the host and the build interpreter being different (if they are the same, of course importlib.machinery.EXTENSION_SUFFIXES needs to be valid for the extension modules being packaged). If the host interpreter is the same as the build interpreter, meson-python already works just fine, AFAIK.

No one is going to use build isolation with cross compilation I think. It'll be something like:

python -m build --wheel --no-isolation --skip-dependency-check -C--cross-file=cross_aarch64_linux.ini

I don't see what going through build and the PEP 517 interfaces gives you in this case. If you need to pass tool-specific command line arguments (the -C--cross-file, which by the way needs to be -Csetup-args=--cross-file=cross_aarch64_linux.ini) you don't even have the advantage of having a tool-agnostic command line interface. Furthermore, you make optional arguments (--no-isolation and --skip-dependency-check) mandatory. It still looks like an hack and a magic incantation more than a solution.

[rgommers]

I think the requirement here is "build interpreter == host interpreter" for now

The problem raised in this issue is about the host and the build interpreter being different

I meant "the same kind, so both CPython or both PyPy". That's a reasonable default, and I think conda-forge's cross compiling jobs for PyPy do that.

I don't see what going through build and the PEP 517 interfaces gives you in this case.

It's the only way to get the .dist-info metadata and a wheel format output that you need. I keep on seeing this confusion, but --no-build-isolation is not niche, it's extremely important (most non-PyPI packagers need this, and I certainly use it more often than not also for local development) and we should treat it on par with the default.

Furthermore, you make optional arguments (--no-isolation and --skip-dependency-check) mandatory. It still looks like an hack and a magic incantation more than a solution.

They're already mandatory for many use cases. --no-isolation was a choice for a default that optimized for "build me a wheel to distribute on PyPI". Many/most other use cases require no isolation. It is definitely not a hack.

[dnicolodi]

It's the only way to get the .dist-info metadata and a wheel format output that you need.

The PEP 517 backend is responsible for generating the .dist-info https://github.com/mesonbuild/meson-python/blob/f9dc18f85475e80e4cba31105cbe6a4e42660b78/mesonpy/__init__.py#L556-L568 It has nothing to do with using a PEP 517 fronend to invoke the wheel packaging tool.

[rgommers]

@dnicolodi I'm not sure what you mean there. meson-python is that backend, and the only way to use meson-python is via a frontend like pip or build.

to invoke the wheel packaging tool.

It seems like you have a conceptual model here that I do not understand. If I understand you correctly, you have something other than Meson and meson-python in mind, but I don't know what that would be.

[eli-schwartz]

From my perspective only have two things: Meson as the build system, and meson-python as the layer between pip & co to build sdists and wheels.

My perspective is a bit different, but I think that's because I approach the whole issue from a different direction

I view meson-python as two things:

• a producer of dist-info metadata, and python distribution (the keyword for "importable thing with metadata) constructor
• binary archiver for pip-compatible installer bundles

And you mention "pip & co" but I think it's a bit simpler, and reduces down to "pip". Or maybe "PyPI".

As I alluded to above, for local builds a wheel is a waste of time, and so are platform tags. What you actually want is a site-packages, and wheels are just a way for build backends to tell installers what files to install. System package managers like conda, dpkg, rpm, pacman, portage, etc. don't care about wheels, because they have better formats that provide crucial cross-domain dependency management among other things. They also, consequently, have better ways to define platform tags than, well, using anything as ill-defined and non-granular as platform tags.

And what even looks at platform tags anyway? Just pip, basically... and, importantly, only in the context of downloading from PyPI.

...

From the perspective of another package manager trying to repackage software produced by the pip package manager, wheels look like those makefiles where "make" creates a tar.gz file that you have to untar, and don't provide a "make install" rule.

auditwheel was brought up by both of you, but it really isn't relevant. Its job is specifically to deal with distributing wheels on PyPI. There are many packaging systems other than PyPI, so we cannot rely on auditwheel for anything.

But this does in fact tie into my belief that platform tags are also specifically to deal with distributing wheels on PyPI.

The rest of the time it is a vestigial organ. While it can't hurt to get it correct where possible, this shouldn't come at the sacrifice of important functionality like generating a successful build+install. When in doubt, apply either a validating placeholder or a genetic tag that is technically correct but provides no information, like "linux" (is that a valid tag? Do you need the CPU architecture?)

The result doesn't matter, if you're locally building the wheel, locally installing it, and locally creating a conda package out of it.

I don't see what going through build and the PEP 517 interfaces gives you in this case. [...] you don't even have the advantage of having a tool-agnostic command line interface.

Because setuptools install had the implementation behavior of a) executing easy_install instead of pip, b) producing egg-info instead of dist-info, and made the unusual development decision to claim that they can't change this because projects might be depending on egg-info specifically, therefore "we will make you stop using egg-info by using bdist_wheel, dropping install, and breaking your project anyway". They then went all-in and declared that they were removing support for interacting with setuptools via a command line.

Because of the privileged position setuptools held in the ecosystem, this has become the new model for build backends, namely, that they shouldn't provide a command line. And ex post facto, this has been reinvented, rather than being due to peculiarities of egg-info, to instead be due to a host of imagined reasons for why command lines are bad, even as an alternative. ;)

The result is that the advantage you get from going via build and a series of command-line arguments is "it's a program that can generate a library API call to a build backend library".

Flit is fighting this trend, as flit_core.wheel provides a command line. However I think the main motivation there is to make it easily bootstrappable (you don't need to build build without having build, before you can build flit_core), not about how noisy the command line is.

It's a weird quirk but ultimately not a huge deal now that the core infrastructure has to some degree settled on flit.

Note that any program which uses PEP 517 to generate a library API call to build a wheel, is a PEP 517 frontend. But not all PEP 517 frontends support build isolation, or default to it. For example, Gentoo Linux has written an internal frontend called gpep517, which is a "Gentoo pep517", that relies on the knowledge that Gentoo never ever wants build isolation.

Yes, we're now seeing a proliferation of incompatible frontend command lines as a retaliatory response to the unification of backend APIs. And no, frontends aren't trivial to write either.

[rgommers]

As I alluded to above, for local builds a wheel is a waste of time, and so are platform tags

Yes and no. All you do with the final zipfile is unpack it and throw it away, so from that perspective it doesn't do much. However, you do need the metadata installed. So a meson install into site-packages gets you a working package, but you're missing the metadata needed to uninstall again. A wheel in this respect is like a filter that ensures everything one needs is present, so uninstalling/upgrading with pip afterwards works, as do things like using importlib.resources.

System package managers like conda, dpkg, rpm, pacman, portage, etc. don't care about wheels

Not as a distribution format, but they do in practice. These tools very often run pip install . --no-build-isolation to build the package. And then as a final step repackage it into a .conda, .rpm or whatever their native format is.

They then went all-in and declared that they were removing support for interacting with setuptools via a command line.

I completely agree that that's a mistake, and I appreciate Meson's nice CLI. But I think that's a mostly unrelated point here. If meson had a command that'd yield the exact same result as pip install . --no-build-isolation then I'd say we could use that and there'd be no need to go through a wheel. But there's no such command (yet, at least).

[eli-schwartz]

Yes and no. All you do with the final zipfile is unpack it and throw it away, so from that perspective it doesn't do much. However, you do need the metadata installed. So a meson install into site-packages gets you a working package, but you're missing the metadata needed to uninstall again. A wheel in this respect is like a filter that ensures everything one needs is present, so uninstalling/upgrading with pip afterwards works, as do things like using importlib.resources.

Right, like I said, meson-python does two things, and one of them is producing that metadata, and the other one is producing that wheel.

I completely agree that that's a mistake, and I appreciate Meson's nice CLI. But I think that's a mostly unrelated point here.

Right, this is very much a side topic in response to @dnicolodi's question about "I don't see what going through build and the PEP 517 interfaces gives you in this case" and magic incantations.

[FFY00]

They then went all-in and declared that they were removing support for interacting with setuptools via a command line.

I completely agree that that's a mistake

There were very good reasons for that, btw. And deprecating is different than removing, currently there's no plan to drop support for invoking setup.py directly.

If meson had a command that'd yield the exact same result as pip install . --no-build-isolation then I'd say we could use that and there'd be no need to go through a wheel. But there's no such command (yet, at least).

I essentially proposed this in https://github.com/mesonbuild/meson/issues/11462.

[FFY00]

importlib.machinery.EXTENSION_SUFFIXES here is only as a safety check. We can remove it. But I would be very surprised if there are no other things that break.

I think the best action here is to skip the check when cross compiling.

[eli-schwartz]

There were very good reasons for that, btw. And deprecating is different than removing, currently there's no plan to drop support for invoking setup.py directly.

I really don't think there was. If my PR to enhance python -m setuptools.launch (an existing functionality) had been accepted, it could have been built upon to provide:

• reading of setup.cfg setup_requires, before evaluating setup.py
• reading of pyproject.toml build-system.requires, before evaluating setup.py
• reimplementing setup.py install to be based on setup.py bdist_wheel instead of setup.py install_egg_info, or even just copying it to setup.py bdist_install.

(The first two of these are the classic issues brought up for why a setuptools command line is inherently bad, and they're the easiest ones to solve, too. In general, I agree that they're all worth solving. Ultimately they ended up being solved via deprecation, rather than via making it work.)

Since there was active disinterest in this, I stopped arguing. The current methods have viable handling. There were unexplored alternative options, but it is what it is, at this point. :)

[FFY00]

That issue has already been discussed at length in the proper places, so I am not gonna repeat that discussion here. Your proposal had nothing to do with avoiding the deprecation of setup.py calls.

[eli-schwartz]

I... don't think I ever did say any such thing???

I did say that I didn't bother attempting to make any additional proposals, since the one I did make was rejected on the grounds that the entire topic of setup.py calls was deprecated and there was no interest in un-deprecating it.

[FFY00]

As a reply to me saying that direct setup.py invocations were deprecated for good reasons, you said you didn't think so and then mentioned your proposal, so I read it as you were trying to say it was somehow meant to fix things. But I guess it was meant to say CLI support could be kept? Anyway, the setuptools maintainers decided that invoking setup.py was deprecated, and that they didn't want to support a new CLI[^1] in favor of 3rd party PEP 517 frontends. I think this makes sense, because it'd be even more confusing to have yet another tool-specific CLI, while we are pushing for standardized tools, and that CLI would only be available on newer setuptools versions, while PEP 517 just work everywhere.

This isn't very relevant here anyway.

[^1]: Your proposal was a new CLI that used setuptools.launch internally, and setuptools.launch was never even meant to be used like that (see https://setuptools.pypa.io/en/latest/history.html#id1213), so yes, I am considering a new CLI, it'd be a new functionality.

[rgommers]

I still haven't figured out what the best way is to test cross-compiling in CI. Getting a Docker image with a cross-compilation toolchain is easy, the annoying part is that we need a host (non-native) Python installed, which isn't common. If anyone has a good idea for anything off-the-shelf, I'd be all ears.

[isuruf]

conda-forge?

[rgommers]

@isuruf yes that was an idea I had, and of course I'd like that (it's my comfort zone:)). I just couldn't figure out just yet how to stop at the right point, with the non-native env prepared with dependencies. I tried in a clone of scipy-feedstock things like:

export BUILD_WITH_CONDA_DEBUG=1
python build-locally.py  # select x86-64 -> aarch64 build

Not sure if it's feasible to do anything like that. Or should I be writing a new Dockerfile which installs two conda envs from scratch, one non-native? The cross build preparation seems to be done inside conda-build, so accessing it wasn't completely obvious to me.

[isuruf]

It's easier to set up the two environments outside of conda-build.

For eg to cross compile for linux-aarch64

conda create -n build-env gxx_linux-aarch64 gfortran_linux-aarch64 python numpy
CONDA_SUBDIR=linux-aarch64 conda create -n host-env python numpy openblas libstdcxx-ng libgfortran-ng
export PREFIX=/path/to/host-env
export BUILD_PREFIX=/path/to/build-env
export CONDA_BUILD=1
conda activate build-env

[rgommers]

Thanks @isuruf! CONDA_SUBDIR= was the piece of the puzzle I was missing.

[doronbehar]

Hey all :) I'm struggling for a while now with cross compiling scipy with Nix. I haven't read the whole discussion, but I just wish to disable this _stable_abi check as suggested earlier. Can anyone guide me how to disable it? With a patch to meson-python of course...

[rgommers]

Hey @doronbehar, carrying gh-322 as a patch should get you across that hurdle. At least for conda-forge that was all that was needed to get cross-compilation to work. They use crossenv, and other distros typically also do some manual patching. For SciPy itself you probably want to use https://github.com/scipy/scipy/pull/18034, which is now documented in http://scipy.github.io/devdocs/dev/contributor/meson_advanced.html#cross-compilation. Please feel free to ping me on a Nix PR/issue if needed.

[rgommers]

I'll cross-link https://github.com/mesonbuild/meson/issues/7049#issuecomment-1476022775, which I think is the key issue to figure out cross-compilation better. There's nothing in meson-python to do at the moment AFAIK, things should work.

[zboszor]

FWIW, I have a working Yocto bbclass and a python3-meson-python recipe that builds the native variant of meson-python and I have a python3-scikit-image 0.20.0 recipe as well which uses build-backend = mesonpy. The cross-compiler tricks for Yocto were:

• pass meson options via --config-setting=setup-args="..."
• patch meson-python so only the manually passed-in setup-args value is used and
• the string value is split into a list so separate arguments in the same string are passed as separate arguments to meson

[rgommers]

Thanks for sharing @zboszor! I had a look, and I think it's this thread?

patch meson-python so only the manually passed-in setup-args value is used and

This was the one that sounded like we needed to fix something in meson-python, but I don't see it from that patch. I think you're referring to disabling build isolation or other CLI flags that you must use to build for Yocto?

[zboszor]

Thanks for sharing @zboszor! I had a look, and I think it's this thread?

patch meson-python so only the manually passed-in setup-args value is used and

This was the one that sounded like we needed to fix something in meson-python, but I don't see it from that patch. I think you're referring to disabling build isolation or other CLI flags that you must use to build for Yocto?

Specifically, this patch.

[dnicolodi]

@zboszor Can you please elaborate on what that is the intent of that patch and why you need it?

If you need to overwrite arguments passed to meson setup, you can just add them to the setup-args: user arguments are passed after the default ones and thus take precedence. One thing that the patch does is to remove the --pefix argument to meson setup, thus when meson-python executes meson install the files are installed into the system python path, to be then copied into the wheel. I'm not sure this is desirable.

Another problematic thing that the path does is this:

-        r = subprocess.run(list(args), env=self._env, cwd=self._build_dir)
+        r = subprocess.run(' '.join(list(args)).split(), env=self._env, cwd=self._build_dir)

which is pointless at best (when args does not contain any string containing white space).

[rgommers]

I was also puzzled with that patch - I'd think none of that should be necessary, and agree with @dnicolodi's suggestions.

Here is the path in full in formatted form, for ease of reading for others:

```diff diff --git a/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch b/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch new file mode 100644 index 000000000..3edda85dc --- /dev/null +++ b/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch @@ -0,0 +1,123 @@ --- meson_python-0.13.0.pre0/mesonpy/__init__.py.old 1970-01-01 01:00:00.000000000 +0100 +++ meson_python-0.13.0.pre0/mesonpy/__init__.py 2023-03-13 21:26:52.263117416 +0100 @@ -669,8 +669,6 @@ self._build_dir = pathlib.Path(build_dir).absolute() if build_dir else (self._working_dir / 'build') self._editable_verbose = editable_verbose self._install_dir = self._working_dir / 'install' - self._meson_native_file = self._source_dir / '.mesonpy-native-file.ini' - self._meson_cross_file = self._source_dir / '.mesonpy-cross-file.ini' self._meson_args: MesonArgs = collections.defaultdict(list) self._env = os.environ.copy() @@ -679,32 +677,6 @@ if ninja_path is not None: self._env.setdefault('NINJA', str(ninja_path)) - # setuptools-like ARCHFLAGS environment variable support - if sysconfig.get_platform().startswith('macosx-'): - archflags = self._env.get('ARCHFLAGS') - if archflags is not None: - arch, *other = filter(None, (x.strip() for x in archflags.split('-arch'))) - if other: - raise ConfigError(f'Multi-architecture builds are not supported but $ARCHFLAGS={archflags!r}') - macver, _, nativearch = platform.mac_ver() - if arch != nativearch: - x = self._env.setdefault('_PYTHON_HOST_PLATFORM', f'macosx-{macver}-{arch}') - if not x.endswith(arch): - raise ConfigError(f'$ARCHFLAGS={archflags!r} and $_PYTHON_HOST_PLATFORM={x!r} do not agree') - family = 'aarch64' if arch == 'arm64' else arch - cross_file_data = textwrap.dedent(f''' - [binaries] - c = ['cc', '-arch', {arch!r}] - cpp = ['c++', '-arch', {arch!r}] - [host_machine] - system = 'Darwin' - cpu = {arch!r} - cpu_family = {family!r} - endian = 'little' - ''') - self._meson_cross_file.write_text(cross_file_data) - self._meson_args['setup'].extend(('--cross-file', os.fspath(self._meson_cross_file))) - # load config -- PEP 621 support is optional self._config = tomllib.loads(self._source_dir.joinpath('pyproject.toml').read_text()) self._pep621 = 'project' in self._config @@ -749,19 +721,6 @@ [binaries] python = '{sys.executable}' ''') - native_file_mismatch = ( - not self._meson_native_file.exists() - or self._meson_native_file.read_text() != native_file_data - ) - if native_file_mismatch: - try: - self._meson_native_file.write_text(native_file_data) - except OSError: - # if there are permission errors or something else in the source - # directory, put the native file in the working directory instead - # (this won't survive multiple calls -- Meson will have to be reconfigured) - self._meson_native_file = self._working_dir / '.mesonpy-native-file.ini' - self._meson_native_file.write_text(native_file_data) # Don't reconfigure if build directory doesn't have meson-private/coredata.data # (means something went wrong) @@ -784,7 +743,7 @@ def _proc(self, *args: str) -> None: """Invoke a subprocess.""" print('{cyan}{bold}+ {}{reset}'.format(' '.join(args), **_STYLES)) - r = subprocess.run(list(args), env=self._env, cwd=self._build_dir) + r = subprocess.run(' '.join(list(args)).split(), env=self._env, cwd=self._build_dir) if r.returncode != 0: raise SystemExit(r.returncode) @@ -800,22 +759,6 @@ """ sys_paths = mesonpy._introspection.SYSCONFIG_PATHS setup_args = [ - f'--prefix={sys.base_prefix}', - os.fspath(self._source_dir), - os.fspath(self._build_dir), - f'--native-file={os.fspath(self._meson_native_file)}', - # TODO: Allow configuring these arguments - '-Ddebug=false', - '-Doptimization=2', - - # XXX: This should not be needed, but Meson is using the wrong paths - # in some scenarios, like on macOS. - # https://github.com/mesonbuild/meson-python/pull/87#discussion_r1047041306 - '--python.purelibdir', - sys_paths['purelib'], - '--python.platlibdir', - sys_paths['platlib'], - # user args *self._meson_args['setup'], ] @@ -905,8 +848,7 @@ editable_verbose: bool = False, ) -> Iterator[Project]: """Creates a project instance pointing to a temporary working directory.""" - with tempfile.TemporaryDirectory(prefix='.mesonpy-', dir=os.fspath(source_dir)) as tmpdir: - yield cls(source_dir, tmpdir, build_dir, meson_args, editable_verbose) + yield cls(source_dir, build_dir, build_dir, meson_args, editable_verbose) @functools.lru_cache() def _info(self, name: str) -> Dict[str, Any]: @@ -1105,15 +1047,7 @@ for key, value in config_settings.items() } - builddir_value = config_settings.get('builddir', {}) - if len(builddir_value) > 0: - if len(builddir_value) != 1: - raise ConfigError('Only one value for configuration entry "builddir" can be specified') - builddir = builddir_value[0] - if not isinstance(builddir, str): - raise ConfigError(f'Configuration entry "builddir" should be a string not {type(builddir)}') - else: - builddir = None + builddir = os.environ.get('MESONPY_BUILD') def _validate_string_collection(key: str) -> None: assert isinstance(config_settings, Mapping) ```

[zboszor]

@zboszor Can you please elaborate on what that is the intent of that patch and why you need it?

If you need to overwrite arguments passed to meson setup, you can just add them to the setup-args: user arguments are passed after the default ones and thus take precedence. One thing that the patch does is to remove the --pefix argument to meson setup, thus when meson-python executes meson install the files are installed into the system python path, to be then copied into the wheel. I'm not sure this is desirable.

This patch is strictly Yocto-specific. The whole meson cross-compiler environment is provided by a meson.bbclass with the native and cross files already created.

The variables passed to meson change from package to package, i.e. the build prefix is package-specific.

It makes little sense to patch pyproject.toml in every package that would use build-backend = mesonpy, because it would mean adapting the patch whenever something changes in this file and the patch may have rejects or fuzz, both are errors in Yocto.

Therefore, passing setup-args on the command line is easier and easier to maintain.

Another problematic thing that the path does is this:

-        r = subprocess.run(list(args), env=self._env, cwd=self._build_dir)
+        r = subprocess.run(' '.join(list(args)).split(), env=self._env, cwd=self._build_dir)

which is pointless at best (when args does not contain any string containing white space).

But it does contain spaces, because there are multiple options. See the the python_mesonpy.bbclass. meson.bbclass also passes these options to meson in the same order.

PEP517_BUILD_OPTS = '--config-setting=setup-args="${MESONOPTS} ${MESON_SOURCEPATH} ${B} ${MESON_CROSS_FILE} ${EXTRA_OEMESON}"'

Without splitting args here, the whole argument is interpreted as a single string and passed to meson as a single command line argument. meson then tries to interpret the complete string as either source or build directly, which fails.

[zboszor]

I was also puzzled with that patch - I'd think none of that should be necessary, and agree with @dnicolodi's suggestions.

Here is the path in full in formatted form, for ease of reading for others:

diff --git a/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch b/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch
new file mode 100644
index 000000000..3edda85dc
--- /dev/null
+++ b/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch
@@ -0,0 +1,123 @@
--- meson_python-0.13.0.pre0/mesonpy/__init__.py.old  1970-01-01 01:00:00.000000000 +0100
+++ meson_python-0.13.0.pre0/mesonpy/__init__.py  2023-03-13 21:26:52.263117416 +0100
@@ -669,8 +669,6 @@
         self._build_dir = pathlib.Path(build_dir).absolute() if build_dir else (self._working_dir / 'build')
         self._editable_verbose = editable_verbose
         self._install_dir = self._working_dir / 'install'
-        self._meson_native_file = self._source_dir / '.mesonpy-native-file.ini'
-        self._meson_cross_file = self._source_dir / '.mesonpy-cross-file.ini'

These files are not needed, Yocto creates its own native and cross files.

         self._meson_args: MesonArgs = collections.defaultdict(list)
         self._env = os.environ.copy()

@@ -679,32 +677,6 @@
         if ninja_path is not None:
             self._env.setdefault('NINJA', str(ninja_path))

-        # setuptools-like ARCHFLAGS environment variable support
-        if sysconfig.get_platform().startswith('macosx-'):
-            archflags = self._env.get('ARCHFLAGS')
-            if archflags is not None:
-                arch, *other = filter(None, (x.strip() for x in archflags.split('-arch')))
-                if other:
-                    raise ConfigError(f'Multi-architecture builds are not supported but $ARCHFLAGS={archflags!r}')
-                macver, _, nativearch = platform.mac_ver()
-                if arch != nativearch:
-                    x = self._env.setdefault('_PYTHON_HOST_PLATFORM', f'macosx-{macver}-{arch}')
-                    if not x.endswith(arch):
-                        raise ConfigError(f'$ARCHFLAGS={archflags!r} and $_PYTHON_HOST_PLATFORM={x!r} do not agree')
-                    family = 'aarch64' if arch == 'arm64' else arch
-                    cross_file_data = textwrap.dedent(f'''
-                        [binaries]
-                        c = ['cc', '-arch', {arch!r}]
-                        cpp = ['c++', '-arch', {arch!r}]
-                        [host_machine]
-                        system = 'Darwin'
-                        cpu = {arch!r}
-                        cpu_family = {family!r}
-                        endian = 'little'
-                    ''')
-                    self._meson_cross_file.write_text(cross_file_data)
-                    self._meson_args['setup'].extend(('--cross-file', os.fspath(self._meson_cross_file)))

This chunk is indeed not needed, Yocto builds under Linux.

         # load config -- PEP 621 support is optional
         self._config = tomllib.loads(self._source_dir.joinpath('pyproject.toml').read_text())
         self._pep621 = 'project' in self._config
@@ -749,19 +721,6 @@
             [binaries]
             python = '{sys.executable}'
         ''')
-        native_file_mismatch = (
-            not self._meson_native_file.exists()
-            or self._meson_native_file.read_text() != native_file_data
-        )
-        if native_file_mismatch:
-            try:
-                self._meson_native_file.write_text(native_file_data)
-            except OSError:
-                # if there are permission errors or something else in the source
-                # directory, put the native file in the working directory instead
-                # (this won't survive multiple calls -- Meson will have to be reconfigured)
-                self._meson_native_file = self._working_dir / '.mesonpy-native-file.ini'
-                self._meson_native_file.write_text(native_file_data)

This chunk is needed. The file mismatch handling doesn't work particularly well because the native file is fed externally. The internally generated cross and native files lines were removed for the same reason.

         # Don't reconfigure if build directory doesn't have meson-private/coredata.data
         # (means something went wrong)
@@ -784,7 +743,7 @@
     def _proc(self, *args: str) -> None:
         """Invoke a subprocess."""
         print('{cyan}{bold}+ {}{reset}'.format(' '.join(args), **_STYLES))
-        r = subprocess.run(list(args), env=self._env, cwd=self._build_dir)
+        r = subprocess.run(' '.join(list(args)).split(), env=self._env, cwd=self._build_dir)

Already explained, setup-args consists of multiple space separated options, which are passed as a single string.

         if r.returncode != 0:
             raise SystemExit(r.returncode)

@@ -800,22 +759,6 @@
         """
         sys_paths = mesonpy._introspection.SYSCONFIG_PATHS
         setup_args = [
-            f'--prefix={sys.base_prefix}',

--prefix is indeed passed in by Yocto's settings.

-            os.fspath(self._source_dir),
-            os.fspath(self._build_dir),

Yocto also passes the source and build directories, with the build directory being outside of the source. What happens if both the source and build directories are specified twice on the meson command line?

-            f'--native-file={os.fspath(self._meson_native_file)}',
-            # TODO: Allow configuring these arguments
-            '-Ddebug=false',
-            '-Doptimization=2',
-
-            # XXX: This should not be needed, but Meson is using the wrong paths
-            #      in some scenarios, like on macOS.
-            #      https://github.com/mesonbuild/meson-python/pull/87#discussion_r1047041306
-            '--python.purelibdir',
-            sys_paths['purelib'],
-            '--python.platlibdir',
-            sys_paths['platlib'],

purelib and platlib are not passed by Yocto and the build can fail because the native python's settings are not what the target build expects.

             # user args
             *self._meson_args['setup'],
         ]
@@ -905,8 +848,7 @@
         editable_verbose: bool = False,
     ) -> Iterator[Project]:
         """Creates a project instance pointing to a temporary working directory."""
-        with tempfile.TemporaryDirectory(prefix='.mesonpy-', dir=os.fspath(source_dir)) as tmpdir:
-            yield cls(source_dir, tmpdir, build_dir, meson_args, editable_verbose)
+        yield cls(source_dir, build_dir, build_dir, meson_args, editable_verbose)

The build directory is created in advance and is different from what meson-python expects here. This is why it's fed using an environment variable.

     @functools.lru_cache()
     def _info(self, name: str) -> Dict[str, Any]:
@@ -1105,15 +1047,7 @@
         for key, value in config_settings.items()
     }

-    builddir_value = config_settings.get('builddir', {})
-    if len(builddir_value) > 0:
-        if len(builddir_value) != 1:
-            raise ConfigError('Only one value for configuration entry "builddir" can be specified')
-        builddir = builddir_value[0]
-        if not isinstance(builddir, str):
-            raise ConfigError(f'Configuration entry "builddir" should be a string not {type(builddir)}')
-    else:
-        builddir = None
+    builddir = os.environ.get('MESONPY_BUILD')

     def _validate_string_collection(key: str) -> None:
         assert isinstance(config_settings, Mapping)

I hope I explained everything.

I can agree that none of this is needed for a straight build on the host which is also the target system, but Yocto's cross-compiler system needs them.

[eli-schwartz]

• pretty sure you're supposed to pass -Csetup-args=--foo -Csetup-args=--bar so that they are parsed as an array, specifically to avoid string parsing as shell code, which inherently doesn't work because your patch is terminally broken :) it assumes that string.split() is a valid way to tokenize a command line into words. That being said, if yocto happens to know that a yocto bbclass will only use the subset of shell words that are also whitespace-splitted words, this is "safe" (and even more yocto-specific than before). I guess this is awkward because the mesonopts abstraction happens at a more fundamental level than the use of meson-python, and isn't a bash array (??) so it's not trivial to loop over it and do for x in "${opts[@]}"; do newopts+=("-Csetup-args=$x"); done
• the recommended way to set the builddir at the moment is -Cbuilddir=... rather than "try -Csetup-args=... and have meson-python iterate over every setup argument, import meson's argument parser which isn't public API, and determine whether it represents an operand rather than one half of an option-argument"

[zboszor]

Thanks, I will try these.

Still, meson-python shouldn't set --python.purelibdir and --python.platlibdir because that does break cross-compiling.

[dnicolodi]

Still, meson-python shouldn't set --python.purelibdir and --python.platlibdir because that does break cross-compiling.

How does passing this arguments break cross compilation? They only tell Meson where to install some files. Then meson-python picks the files up from the specified location and packs them into a wheel. The content of the wheel does not depend in any way on these arguments.

[dnicolodi]

@zboszor You seem to be patching meson-python for the wrong reasons.

These files are not needed, Yocto creates its own native and cross files.

You can have as many native and cross files as you want. Meson merges them. You don't need to remove the arguments passed to Meson by meson-python to pass your own cross and native files.

This chunk is needed. The file mismatch handling doesn't work particularly well because the native file is fed externally.

If you leave the meson-python generated native file in place, you don't need to remove this code. However, this code will be gone in the next meson-python version.

Already explained, setup-args consists of multiple space separated options, which are passed as a single string.

This is an horrible idea, as @eli-schwartz explained. If you need to pass multiple arguments via setup-args you need to pass them as list. pip till the latest version does not allow this. There is a patch already merged that fixes this, though. If you need to use pip, you can most likely apply that patch to the pip version you are using. pypa/build allows to pass a list of options to setup-args using the -C command line argument multiple times:

python -m build -Csetup-args=-Cfoo=bar -Csetup-args=debug=true -Cbuilddir=/tmp/build3452

--prefix is indeed passed in by Yocto's settings.

The argument to the --prefix setup option should not have any effect on the content of the wheel.

Yocto also passes the source and build directories, with the build directory being outside of the source.

You can pass the build directory to use to meson-python, see the example above. The source directory is passed to meson-python by the Python build front-end (pypa/build or pip or whatever) there is no need to pass it separately.

purelib and platlib are not passed by Yocto and the build can fail because the native python's settings are not what the target build expects.

As already explained, these settings should not have any effect on the content of the generated wheel.

I can agree that none of this is needed for a straight build on the host which is also the target system, but Yocto's cross-compiler system needs them.

I think that none of this is required, the patch is probably based on a misunderstanding of how meson-python works.

[zboszor]

Thank you for all the suggestions and constructive criticism. The new series is here now, the meson-python recipe doesn't need any patches: python_mesonpy.bbclass meson-python recipe