meson: build fails with python 3.12

davidak commented 1 year ago

Describe the bug

error: builder for '/nix/store/mk0py0cfw91phxhihpfgana0pjmqhzpw-meson-1.2.0.drv' failed with exit code 8;

See log

``` ... Executing subproject bar Project name: bar Project version: undefined C++ compiler for the host machine: g++ (gcc 12.3.0 "g++ (GCC) 12.3.0") C++ linker for the host machine: g++ ld.bfd 2.40 C++ compiler for the build machine: g++ (gcc 12.3.0 "g++ (GCC) 12.3.0") C++ linker for the build machine: g++ ld.bfd 2.40 Could not introspect Python (['/nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python3', '/build/source/mesonbuild/scripts/python_info.py']): exit code 1 Program stdout: Program stderr: Traceback (most recent call last): File "/build/source/mesonbuild/scripts/python_info.py", line 16, in import distutils.command.install ModuleNotFoundError: No module named 'distutils' test cases/python/6 failing subproject/subprojects/bar/meson.build:3:26: Exception: ['/nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python3']> is not a valid python or it is missing distutils Subproject subprojects/bar is buildable: NO (disabling) Dependency bar from subproject bar found: NO (subproject failed to configure) Could not introspect Python (['/nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python3', '/build/source/mesonbuild/scripts/python_info.py']): exit code 1 Program stdout: Program stderr: Traceback (most recent call last): File "/build/source/mesonbuild/scripts/python_info.py", line 16, in import distutils.command.install ModuleNotFoundError: No module named 'distutils' test cases/python/6 failing subproject/meson.build:5:26: ERROR: ['/nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python3']> is not a valid python or it is missing distutils Traceback (most recent call last): File "/build/source/run_tests.py", line 315, in run_configure_inprocess returncode = mesonmain.run(commandlist, get_meson_script()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/mesonmain.py", line 285, in run return CommandLineParser().run(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/mesonmain.py", line 196, in run return errorhandler(e, command) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/mesonmain.py", line 40, in errorhandler raise e File "/build/source/mesonbuild/mesonmain.py", line 194, in run return options.run_func(options) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/msetup.py", line 358, in run app.generate() File "/build/source/mesonbuild/msetup.py", line 183, in generate return self._generate(env, capture, vslite_ctx) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/msetup.py", line 228, in _generate intr.run() File "/build/source/mesonbuild/interpreter/interpreter.py", line 2997, in run super().run() File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 164, in run self.evaluate_codeblock(self.ast, start=1) File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 190, in evaluate_codeblock raise e File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 182, in evaluate_codeblock self.evaluate_statement(cur) File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 198, in evaluate_statement self.assignment(cur) File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 635, in assignment value = self.evaluate_statement(node.value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 200, in evaluate_statement return self.method_call(cur) ^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 535, in method_call obj = self.evaluate_statement(invocable) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 200, in evaluate_statement return self.method_call(cur) ^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 550, in method_call res = obj.method_call(method_name, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreter/interpreterobjects.py", line 814, in method_call ret = method(state, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/decorators.py", line 109, in wrapped ret = f(*wrapped_args, **wrapped_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/decorators.py", line 277, in wrapper return f(*nargs, **wrapped_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/decorators.py", line 596, in wrapper return f(*wrapped_args, **wrapped_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/modules/python.py", line 437, in find_installation python = self._find_installation_impl(state, display_name, name_or_path, required) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/modules/python.py", line 398, in _find_installation_impl raise mesonlib.MesonException(sanitymsg) mesonbuild.utils.core.MesonException: ['/nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python3']> is not a valid python or it is missing distutils ====================================== python: 7 install path ====================================== (inprocess) $ setup --prefix /usr --libdir lib 'test cases/python/7 install path' '/build/source/b 87a5fe311e' --backend=ninja Build started at 2023-09-07T00:26:10.979266 Main binary: /nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python Build Options: -Dprefix=/usr -Dlibdir=lib -Dbackend=ninja Python system: Linux The Meson build system Version: 1.2.0 Source dir: /build/source/test cases/python/7 install path Build dir: /build/source/b 87a5fe311e Build type: native build Project name: install path Project version: undefined Build machine cpu family: x86_64 Build machine cpu: x86_64 Host machine cpu family: x86_64 Host machine cpu: x86_64 Target machine cpu family: x86_64 Target machine cpu: x86_64 Could not introspect Python (['/nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python', '/build/source/mesonbuild/scripts/python_info.py']): exit code 1 Program stdout: Program stderr: Traceback (most recent call last): File "/build/source/mesonbuild/scripts/python_info.py", line 16, in import distutils.command.install ModuleNotFoundError: No module named 'distutils' test cases/python/7 install path/meson.build:9:22: ERROR: ['/nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python']> is not a valid python or it is missing distutils Traceback (most recent call last): File "/build/source/run_tests.py", line 315, in run_configure_inprocess returncode = mesonmain.run(commandlist, get_meson_script()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/mesonmain.py", line 285, in run return CommandLineParser().run(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/mesonmain.py", line 196, in run return errorhandler(e, command) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/mesonmain.py", line 40, in errorhandler raise e File "/build/source/mesonbuild/mesonmain.py", line 194, in run return options.run_func(options) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/msetup.py", line 358, in run app.generate() File "/build/source/mesonbuild/msetup.py", line 183, in generate return self._generate(env, capture, vslite_ctx) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/msetup.py", line 228, in _generate intr.run() File "/build/source/mesonbuild/interpreter/interpreter.py", line 2997, in run super().run() File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 164, in run self.evaluate_codeblock(self.ast, start=1) File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 190, in evaluate_codeblock raise e File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 182, in evaluate_codeblock self.evaluate_statement(cur) File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 198, in evaluate_statement self.assignment(cur) File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 635, in assignment value = self.evaluate_statement(node.value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 200, in evaluate_statement return self.method_call(cur) ^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/interpreterbase.py", line 550, in method_call res = obj.method_call(method_name, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreter/interpreterobjects.py", line 814, in method_call ret = method(state, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/decorators.py", line 109, in wrapped ret = f(*wrapped_args, **wrapped_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/decorators.py", line 277, in wrapper return f(*nargs, **wrapped_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/interpreterbase/decorators.py", line 596, in wrapper return f(*wrapped_args, **wrapped_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/modules/python.py", line 437, in find_installation python = self._find_installation_impl(state, display_name, name_or_path, required) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/source/mesonbuild/modules/python.py", line 398, in _find_installation_impl raise mesonlib.MesonException(sanitymsg) mesonbuild.utils.core.MesonException: ['/nix/store/q5fi2vmnii9vjpqynp56vp0734yhpaig-python3-3.12.0b4/bin/python']> is not a valid python or it is missing distutils ```

Steps To Reproduce

Steps to reproduce the behavior:

nix-shell -I nixpkgs=./ -p python312Packages.meson

Expected behavior

build successful

Additional context

the python 3.12 version of meson is not shown in the search. maybe this is known not to work. i don't see it marked as broken

https://search.nixos.org/packages?channel=unstable&show=python311Packages.meson&from=0&size=50&sort=relevance&type=packages&query=meson

Notify maintainers

@AndersonTorres

Metadata

nixpkgs @ 48992ab9b6173baa5a39f93d06c81f22c42920e8

system: "x86_64-linux"
host os: Linux 6.3.8, NixOS, 22.11 (Raccoon), 22.11.4588.93fddcf640c
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.11.1
channels(root): "home-manager-22.11.tar.gz, nixos-22.11, nixos-hardware, nixos-unstable"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

ConnorBaker commented 1 year ago

EDIT:

Meson tracking issue: https://github.com/mesonbuild/meson/issues/7702
Meson PR: https://github.com/mesonbuild/meson/pull/11133#issuecomment-1627705222

dnicolodi commented 1 year ago

The released meson version depends on distutils which has been removed from Python 3.12. The next meson version will remove the dependency on distutils. setuptools embeds a copy of distutils, thus, until the next meson version is released, you may want to install setuptools on Python 3.12 to get the distutils module.

haampie commented 1 year ago

Installing setuptools is not enough, you have to import it too to let setuptools mess with sys.path s.t. import disutils works.

So, meson should import setuptools before importing disutils if distutils fails to import directly, and meson doesn't do this

dnicolodi commented 1 year ago

No, it is not necessary to import setuptools. setuptools have some magic (details provided on request) to inject its distutils copy in the Python path. The solution I proposed is tested. Do you have any evidence that it does not work?

haampie commented 1 year ago

I'm not an expert on how nix deals with python package installs, but the setuptools hack where sys.path is modified through

<setuptools prefix>/lib/pythonX.Y/site-packages/distutils-precedence.pth

at startup only works when setuptools prefix == python prefix. That is, setuptools must be installed as a system site package.

That assumption may not hold in nix, it certainly does not in the Spack package manager which like nix also puts python packages in their own prefixes, and lets python locate those through PYTHONPATH.

So, if PYTHONPATH is all you have, you need to import setuptools to "activate" the same hack, s.t. import distutils work.

AndersonTorres commented 1 year ago

Should we wait the next release that fixes this?

dwilliam62 commented 4 months ago

Another report of this error with more details. Problem is with meson It has distuils as a dependency That has been removed from Python v3.12 (nice) Have to wait for updated meson or add 'setuputils' in python.

https://github.com/NixOS/nixpkgs/issues/253751

NixOS / nixpkgs