Closed ghost closed 3 years ago
diegoferigo
commented
3 years ago Hi @remy-phelipot, sorry for the late reply, I missed this issue somehow. ninja is a dependency, and it should provide ootb the missing command.
Can you provide more info on your use case?
ghost
commented
3 years ago Hello @diegoferigo.
I tested again and indeed ninja seems to be correctly installed, so my original message do not correctly reflect the actual problem.
I'm encountering the following error when building the example module (I added ninja as build dependency in pyproject.toml to be sure it is installed, but is not necessary):
$ python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (cmake_build_extension, ninja, numpy, pybind11, setuptools>=45, setuptools_scm[toml]>=6.0, wheel)
* Getting dependencies for wheel...
running egg_info
writing src/mymath.egg-info/PKG-INFO
writing dependency_links to src/mymath.egg-info/dependency_links.txt
writing entry points to src/mymath.egg-info/entry_points.txt
writing requirements to src/mymath.egg-info/requires.txt
writing top-level names to src/mymath.egg-info/top_level.txt
writing manifest file 'src/mymath.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
running build_ext
==> Configuring:
$ cmake -S /home/phelipot/projects/cmake-build-extension/example -B /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig -DPython3_ROOT_DIR=/tmp/build-env-j2geah8k -DCALL_FROM_SETUP_PY:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=OFF -DEXAMPLE_WITH_SWIG:BOOL=ON -DEXAMPLE_WITH_PYBIND11:BOOL=OFF
==> Building:
$ cmake --build /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings --config Release
==> Installing:
$ cmake --install /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings
-- Found Python3: /tmp/build-env-j2geah8k/bin/python3 (found version "3.8.10") found components: Interpreter Development.Module NumPy
-- Configuring done
-- Generating done
-- Build files have been written to: /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings
[4/4] Linking CXX shared module bindings_swig/mymath/_swig_bindings.so
-- Install configuration: "Release"
-- Installing: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/libmymath.a
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/include/mymath.h
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/bin/print_answer_swig
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathTargets.cmake
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathTargets-release.cmake
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathConfig.cmake
-- Up-to-date: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/lib/cmake/MyMath/MyMathConfigVersion.cmake
-- Installing: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/bindings.py
-- Installing: /home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig/_swig_bindings.so
==> Configuring:
$ cmake -S /home/phelipot/projects/cmake-build-extension/example -B /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11 -DPython3_ROOT_DIR=/tmp/build-env-j2geah8k -DCALL_FROM_SETUP_PY:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=OFF -DEXAMPLE_WITH_SWIG:BOOL=OFF -DEXAMPLE_WITH_PYBIND11:BOOL=ON
==> Building:
$ cmake --build /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings --config Release
==> Installing:
$ cmake --install /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings
CMake Error at CMakeLists.txt:2 (project):
Running
'/tmp/pip-build-env-rr5z5dv1/overlay/bin/ninja' '--version'
failed with:
No such file or directory
-- Configuring incomplete, errors occurred!
See also "/home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
return self._build_with_temp_dir(['bdist_wheel'], '.whl',
File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
self.run_setup()
File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 43, in <module>
setuptools.setup(
File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/cmake_build_extension/build_extension.py", line 110, in run
self.build_extension(ext)
File "/tmp/build-env-j2geah8k/lib/python3.8/site-packages/cmake_build_extension/build_extension.py", line 224, in build_extension
subprocess.check_call(configure_command)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-S', '/home/phelipot/projects/cmake-build-extension/example', '-B', '/home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_Pybind11Bindings', '-GNinja', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX:PATH=/home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_pybind11', '-DPython3_ROOT_DIR=/tmp/build-env-j2geah8k', '-DCALL_FROM_SETUP_PY:BOOL=ON', '-DBUILD_SHARED_LIBS:BOOL=OFF', '-DEXAMPLE_WITH_SWIG:BOOL=OFF', '-DEXAMPLE_WITH_PYBIND11:BOOL=ON']' returned non-zero exit status 1.
ERROR Backend subproccess exited when trying to invoke build_wheelIf I produce the sdist and the wheel using python -m build, then the build process is executed successfully.
Do you know what is going on and if there is any solution?
diegoferigo
commented
3 years ago Calling python -m build triggers the generation of the sdist and, from it, the wheel. Passing the --wheel option skips the sdist generation and the wheel is built directly from the sources. In your case, there's a mismatch in the two behaviors.
I couldn't reproduce your problem on my PC (see details below). The example has two CMakeExtensions, one for SWIG and the other for pybind11. What's strange is that the former works, but the process fails when the latter is configured. It's odd that it complains about ninja, since it has already successfully built the first extension in the very same isolated environment.
Let's keep investigating, I suspect that is something specific to your setup since it's not happening in mine nor in CI that builds installs and tests in all major OSs.
ghost
commented
3 years ago I reset my environment and do:
$ virtualenv -p python3 .venv
$ . .venv/bin/activate
(.venv) $ pip install --upgrade pip build
(.venv) $ python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (cmake_build_extension, ninja, numpy, pybind11, setuptools>=45,
...
Successfully built mymath-0.0.0-cp38-cp38-linux_x86_64.whlSo far so good. But if I execute the command again it fails:
$ python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (cmake_build_extension, ninja, numpy, pybind11, setuptools>=45,
...
==> Installing:
$ cmake --install /home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings
CMake Error at CMakeLists.txt:2 (project):
Running
'/tmp/build-env-y54elrvx/bin/ninja' '--version'
failed with:
No such file or directory
-- Configuring incomplete, errors occurred!
See also "/home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/phelipot/projects/cmake-build-extension/example/.venv/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
return self._build_with_temp_dir(['bdist_wheel'], '.whl',
File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
self.run_setup()
File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 43, in <module>
setuptools.setup(
File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/cmake_build_extension/build_extension.py", line 110, in run
self.build_extension(ext)
File "/tmp/build-env-bil93mdf/lib/python3.8/site-packages/cmake_build_extension/build_extension.py", line 224, in build_extension
subprocess.check_call(configure_command)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-S', '/home/phelipot/projects/cmake-build-extension/example', '-B', '/home/phelipot/projects/cmake-build-extension/example/build/temp.linux-x86_64-3.8_SwigBindings', '-GNinja', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX:PATH=/home/phelipot/projects/cmake-build-extension/example/build/lib.linux-x86_64-3.8/mymath_swig', '-DPython3_ROOT_DIR=/tmp/build-env-bil93mdf', '-DCALL_FROM_SETUP_PY:BOOL=ON', '-DBUILD_SHARED_LIBS:BOOL=OFF', '-DEXAMPLE_WITH_SWIG:BOOL=ON', '-DEXAMPLE_WITH_PYBIND11:BOOL=OFF']' returned non-zero exit status 1.
ERROR Backend subproccess exited when trying to invoke build_wheelIt seems temporary files are produced in the build directory that interfere between each consecutive build. If I delete the directory, then it works again like a charm.
(.venv) $ rm -rf build && python -m build --wheel
...
Successfully built mymath-0.0.0-cp38-cp38-linux_x86_64.whlOw ok I finally understood the problem after your last comment! The CMakeCache.txt file stored in the build folder contains the path to the ninja command, which is installed in a virtualenv in /tmp. The virtualenv is deleted after the first time the wheel is built, but the build folder, being stored in-source, is kept. The second time you build the wheel, the CMakeCache.txt entry does not get updated with the new virtualenv folder in /tmp, therefore it fails.
I suspect that a workaround is passing in your setup.py the following option: f"-DCMAKE_MAKE_PROGRAM={shutil.which('ninja')}". Can you please try to check if it solves the problem?
ghost
commented
3 years ago Yeap your solution works at least on linux platforms 🙂 As the use of ninja is internal of cmake-build-extension and the end user have no idea of what is going on in this tricky case, may I suggest to add this code in the project? For instance here:
diff --git a/src/cmake_build_extension/build_extension.py b/src/cmake_build_extension/build_extension.py
index a27d144..c4667c7 100644
--- a/src/cmake_build_extension/build_extension.py
+++ b/src/cmake_build_extension/build_extension.py
@@ -148,6 +148,7 @@ class BuildExtension(build_ext):
"-GNinja",
f"-DCMAKE_BUILD_TYPE={ext.cmake_build_type}",
f"-DCMAKE_INSTALL_PREFIX:PATH={cmake_install_prefix}",
+ f"-DCMAKE_MAKE_PROGRAM={shutil.which('ninja')}",
]
# Extend the configure arguments with those passed from the extensionYes this was the idea, thanks for trying! Do you have specific reasons to say that this works only on linux platforms? The shutil module should work also on Windows and macOS.
Feel free to open a PR with this change, otherwise I'll take care of it in the next days.
ghost
commented
3 years ago I agree about Windows compatibility, it should be ok.
Hello,
Ninja is currently the only generator supported by cmake-build-extension: https://github.com/diegoferigo/cmake-build-extension/blob/bb836e445b743ace473e1c9a5666cb69e88aa922/src/cmake_build_extension/build_extension.py#L78
Unfortunately it is not installed on all platform and a great benefit of CMake is to be agnostic of the final build tool that is used.
It would be nice to add a configuration option that select a different generator in replacement of Ninja (e.g. Make)