python3 dependency incorrectly references host python when cross-compiling

[pinchartl]

When cross-compiling a project using the python3 dependency (e.g. py3_dep = dependency('python3', required : get_option('pycamera'))), meson incorrectly uses the Python headers from the host if the cross-compilation environment doesn't include Python.

The issue was introduced starting in

commit aa69cf04484309f82d2da64c433539d2f6f2fa82 (HEAD)
Author: Eli Schwartz <eschwartz@archlinux.org>
Date:   Sun Jul 24 20:09:15 2022 -0400

    merge the python dependency from the python module into dependencies

    We have two copies of this code, and the python module one is vastly
    superior, not just because it allows choosing which python executable to
    base itself on. Unify this. Fixes various issues including non-Windows
    support for sysconfig, and pypy edge cases.

Before that commit, meson failed to find the python3 dependency, as shown in meson.log:

Determining dependency 'python3' with pkg-config executable '/home/user/src/buildroot/output/x86-musl/host/bin/pkgconf'
env[PKG_CONFIG_PATH]:
env[PKG_CONFIG_SYSROOT_DIR]: /home/user/src/buildroot/output/x86-musl/host/i586-buildroot-linux-musl/sysroot
env[PKG_CONFIG_LIBDIR]: /home/user/src/buildroot/output/x86-musl/host/i586-buildroot-linux-musl/sysroot/usr/lib/pkgconfig:/home/user/src/buildroot/output/x86-musl/host/i586-buildroot-linux-musl/sysroot/usr/share/pkgconfig
Called `/home/user/src/buildroot/output/x86-musl/host/bin/pkgconf --modversion python3` -> 1
stderr:
Package python3 was not found in the pkg-config search path.
Perhaps you should add the directory containing `python3.pc'
to the PKG_CONFIG_PATH environment variable
Package 'python3', required by 'virtual:world', not found
-----------
Run-time dependency python3 found: NO (tried pkgconfig and sysconfig)

The offending commit introduced multiple issues that got fixed in subsequent commits, so it's difficult to pinpoint the exact cause. The next stable release of meson, 1.1.0, picked the Python headers from the host:

Determining dependency 'python3' with pkg-config executable '/home/user/src/buildroot/output/x86-musl/host/bin/pkgconf'
env[PKG_CONFIG_PATH]: 
env[PKG_CONFIG_SYSROOT_DIR]: /home/user/src/buildroot/output/x86-musl/host/i586-buildroot-linux-musl/sysroot
env[PKG_CONFIG_LIBDIR]: /home/user/src/buildroot/output/x86-musl/host/i586-buildroot-linux-musl/sysroot/usr/lib/pkgconfig:/home/user/src/buildroot/output/x86-musl/host/i586-buildroot-linux-musl/sysroot/usr/share/pkgconfig
Called `/home/user/src/buildroot/output/x86-musl/host/bin/pkgconf --modversion python3` -> 1
stderr:
Package python3 was not found in the pkg-config search path.
Perhaps you should add the directory containing `python3.pc'
to the PKG_CONFIG_PATH environment variable
Package 'python3', required by 'virtual:world', not found
-----------
Running compile:
Working directory:  /home/user/src/libcamera/build/i586-gcc/meson-private/tmp8g3spk0s 
Command line:  /home/user/src/buildroot/output/x86-musl/host/bin/i586-buildroot-linux-musl-g++ -I/usr/include/python3.11 /home/user/src/libcamera/build/i586-gcc/meson-private/tmp8g3spk0s/testfile.cpp -E -P -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -Os -D_FILE_OFFSET_BITS=64 -P -O0 -fpermissive

Code:

        #ifdef __has_include
         #if !__has_include("Python.h")
          #error "Header 'Python.h' could not be found"
         #endif
        #else
         #include <Python.h>
        #endif
Compiler stdout: 

Compiler stderr:
 i586-buildroot-linux-musl-g++: WARNING: unsafe header/library path used in cross-compilation: '-I/usr/include/python3.11'

Run-time dependency python3 found: YES 3.11

The problem persists in the meson master branch.

I have noticed the problem when trying to cross-compile libcamera (https://git.libcamera.org/libcamera/libcamera.git/), and wrote the following minimal reproducer:

# SPDX-License-Identifier: CC0-1.0

project('reproducer', 'c',
    meson_version : '>= 0.57',
    version : '0.0.0',
    license : 'LGPL 2.1+')

py3_dep = dependency('python3', required : false)

To reproduce the issue, you will need a cross-compilation environment without Python. I personally use buildroot, and can provide a configuration file if desired.

system parameters

• This is a cross build
• Operating system: Gentoo Linux
• Python version: 3.11
• meson version: any starting at 1.1.0
• ninja version: not applicable

[pinchartl]

6 months ping :) We've discussed the issue on IRC a while ago, but the discussion then died out.

[kbingham]

Should we be using

• https://pypi.org/project/meson-python/
• https://github.com/mesonbuild/meson-python/

instead here? Will that handle cross compilation too ?

(Though even then that probably doesn't discount the nature of the issue above)

[pinchartl]

I think the python3 dependency should be fixed regardless. Meson shouldn't pick host libraries when cross-compiling and linking to dependency('foo'), for any value of 'foo'.