Meson 1.2.3 broke extension module compilation on GraalPy

[timfel]

Describe the bug As of https://github.com/mesonbuild/meson/commit/3c3caf5163e2efdb2bc6a7089a7f4e0c5d058efb, GraalPy is no longer detected for compiling extension modules. You get an error like

Program python found: YES (/private/var/folders/sw/dkvc97c952sdhydgtm78_j240000gp/T/pip-install-7p7ez5fi/contourpy_d5f9f6b82ed94700b50dee5e1349ffba/testvenv/bin/graalpy)
  Did not find pkg-config by name 'pkg-config'
  Found Pkg-config: NO
  pybind11-config found: YES (/private/var/folders/sw/dkvc97c952sdhydgtm78_j240000gp/T/pip-build-env-ukpfjb8a/overlay/bin/pybind11-config) 2.10.4
  Run-time dependency pybind11 found: YES 2.10.4
  Configuring _build_config.py using configuration
  Run-time dependency python found: NO (tried sysconfig)

  ../src/meson.build:5:10: ERROR: Python dependency not found

  A full log can be found at /private/var/folders/sw/dkvc97c952sdhydgtm78_j240000gp/T/pip-install-ue7li0dy/contourpy_6529c7e13e7746aa84d9b5a9ce191278/.mesonpy-1ztxj8ac/meson-logs/meson-log.txt
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

@eli-schwartz added a branch for PyPy, GraalPy needs that same branch here

To Reproduce graalpy -m pip install contourpy (or anything else that pulls in meson >= 1.2.3)

Expected behavior Expected behaviour is that the extension compiles.

system parameters

• Is this a cross build or just a plain native build (for the same computer)? Plain native build.

• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.) macOS Big Sur.

• what Python version are you using e.g. 3.8.0 GraalPy 23.1.1 from https://github.com/oracle/graalpython/releases/tag/graal-23.1.1

• what meson --version meson 1.2.3

[timfel]

The test for GraalPy can be done analogous to is_pypy: is_graalpy = '__graalpython__' in sys.builtin_module_names

[eli-schwartz]

Meson has never explicitly had code to support GraalPy and I'm not surprised it doesn't work. I'm a bit more surprised that it worked by accident in the past. :)

I'm not opposed to supporting it, but I think if we do we really need a CI test to verify basic functionality, as we do with PyPy.

I'm not sure how best to do that since unlike PyPy, I've never touched GraalPy and it doesn't even seem to be packaged by Linux distros.

[timfel]

I'm not sure how best to do that since unlike PyPy, I've never touched GraalPy and it doesn't even seem to be packaged by Linux distros.

Mostly we tend to recommend pyenv for posix and setup-python when using Github actions. We (currently) haven't looked at what it would take to support other CI services.

[timfel]

A workaround on our end would be to define LIBPYTHON as "" in our sysconfig, but I'm not sure about the implications here. Our CPython C API emulation library is always a shared library, but we simply always load it into the GraalPy process before loading any extensions, so they don't need to link against it.

[eli-schwartz]

Our CPython C API emulation library is always a shared library, but we simply always load it into the GraalPy process before loading any extensions, so they don't need to link against it.

CPython does this where possible too, in current versions. The main issue is that on some operating systems that's not an option, you have to directly link all loadable extensions to the shared libraries they use, even if that library is loaded at the global process level.

In particular, Windows.

The sysconfig key is defined for CPython as the Makefile replacement variable for inserting into python.pc and python-config.sh (not the -embed variants). Its value as defined by the configure script essentially amounts to a platform check, to see whether CPython is being configured on a platform that requires that quirk (e.g. cygwin).

[eli-schwartz]

Mostly we tend to recommend pyenv for posix and setup-python when using Github actions. We (currently) haven't looked at what it would take to support other CI services.

We use Linux container images (fedora, Ubuntu, arch, opensuse...) with a large number of dependencies preinstalled via the Linux package manager, or sometimes manually compiled and installed. See ci/ciimage for how to update those, in combination with .github/workflows/images.yml

The macOS tests use homebrew and the windows tests use azure pipelines, or msys2 or cygwin dedicated yml files. No containers there, sadly, so we have to reinstall dependencies on every CI run.

[timfel]

The sysconfig key is defined for CPython as the Makefile replacement variable for inserting into python.pc and python-config.sh (not the -embed variants). Its value as defined by the configure script essentially amounts to a platform check, to see whether CPython is being configured on a platform that requires that quirk (e.g. cygwin).

Ok, that sounds like we should just define that on macOS and Linux and not on Windows to behave similarly here.

We use Linux container images (fedora, Ubuntu, arch, opensuse...) with a large number of dependencies preinstalled via the Linux package manager, or sometimes manually compiled and installed. See ci/ciimage for how to update those, in combination with .github/workflows/images.yml

The macOS tests use homebrew and the windows tests use azure pipelines, or msys2 or cygwin dedicated yml files. No containers there, sadly, so we have to reinstall dependencies on every CI run.

I will look into adding GraalPy to the tests and open a PR :)

[timfel]

Just to leave an update here:

Defining LIBPYTHON has fixed our builds on GraalPy with meson >=1.2.3, so we're good on that front for now.

I have looked into running the GraalPy tests the same way as the PyPy tests here, and we are currently failing ~40 tests. A bunch of these seem to be related to running coverage and profiling while running meson. We don't fully support these on GraalPy, so I'm considering just skipping them for us. A couple of other tests are trying to change signal behaviour (like changing SA_RESTART) which we also do cannot support since we're running on a JVM. I'll go through the tests, but just in general, would skipping some be accepted?

[eli-schwartz]

The PyPy tests are set up to do two things:

• test that test cases/python/8 different python versions/ can handle running on CPython and compiling extensions for PyPy
• test that PyPy is a suitable sys.executable for meson, by using it as the interpreter to run_tests.py

I'm happy to accept as much or as little as you feel makes sense, so feel free to skip things on GraalPy as long as they aren't skipped for CPython.