macOS: client project builds but fails to install

[cimes-isi]

Thanks for working on this project. It seems to be what I'm looking for so I'm trying it out to build a C++ executable in my project which I use as a subprocess. This should be an even simpler use case than the current examples in that I don't actually need to generate Python bindings.

I'm using the latest version 0.0.7.

I can build:

$ python -m build .
* Creating venv isolated environment...
* Installing packages in isolated environment... (py-build-cmake)
* Getting dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (py-build-cmake)
* Getting dependencies for wheel...
* Building wheel...
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    Python3_EXECUTABLE
    Python3_FIND_REGISTRY
    Python3_FIND_STRATEGY
    Python3_ROOT_DIR
    VERIFY_VERSION

-- Build files have been written to: /private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/build-via-sdist-28_rq46r/edgepipe-0.1.0/.py-build-cmake_cache/cp38-cp38-macosx_10_14_arm64
[ 50%] Building CXX object CMakeFiles/sched-pipeline.dir/sched-pipeline.cpp.o
[100%] Linking CXX executable sched-pipeline
[100%] Built target sched-pipeline
Installing component scheduler_applications
-- Installing: /var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/tmpnmmn4n8e/staging/sched/sched-pipeline
Successfully built edgepipe-0.1.0.tar.gz and edgepipe-0.1.0-cp38-cp38-macosx_10_14_arm64.whl

But installation fails (both normal and editable):

$ pip install .
Processing /Users/cimes/local/fogsys/EdgePipe
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [31 lines of output]
      Traceback (most recent call last):
        File "/Users/cimes/local/fogsys/EdgePipe/venv/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'
      Traceback (most recent call last):
        File "/Users/cimes/local/fogsys/EdgePipe/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
      AttributeError: module 'py_build_cmake.build' has no attribute 'prepare_metadata_for_build_wheel'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/Users/cimes/local/fogsys/EdgePipe/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/Users/cimes/local/fogsys/EdgePipe/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/cimes/local/fogsys/EdgePipe/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 47, in build_wheel
          whl_name = self.build_wheel_in_dir(wheel_directory, tmp_build_dir)
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 183, in build_wheel_in_dir
          self.do_native_cross_cmake_build(tmp_build_dir, staging_dir,
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 211, in do_native_cross_cmake_build
          self.run_cmake(src_dir, staging_dir, metadata, cmake_cfg, cfg.cross,
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 377, in run_cmake
          self.run(configure_cmd, check=True, env=cmake_env)
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 298, in run
          return sp_run(*args, **kwargs)
        File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 516, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['cmake', '-B', '/Users/cimes/local/fogsys/EdgePipe/.py-build-cmake_cache/cp38-cp38-macosx_10_14_arm64', '-S', '/Users/cimes/local/fogsys/EdgePipe/partition', '-D', 'VERIFY_VERSION=0.1.0', '-D', 'Python3_EXECUTABLE:FILEPATH=/Users/cimes/local/fogsys/EdgePipe/venv/bin/python3', '-D', 'Python3_ROOT_DIR:PATH=/Users/cimes/local/fogsys/EdgePipe/venv', '-D', 'Python3_FIND_REGISTRY=NEVER', '-D', 'Python3_FIND_STRATEGY=LOCATION', '-D', 'CMAKE_BUILD_TYPE=RelWithDebInfo']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

It doesn't show the stdout/stderr for the cmake command that failed. My cmake project depends on another package (located with find_package) and links the executable against it, so perhaps the pip install environment is somehow different than the build environment? It's difficult to tell without additional output.

CMakeLists.txt:

cmake_minimum_required(VERSION 3.8)

project(edgepipe VERSION 0.1.0
                 LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
add_compile_options(-Wall -Wextra)

find_package(yaml-cpp REQUIRED)

add_executable(sched-pipeline sched-pipeline.cpp)
target_link_libraries(sched-pipeline PRIVATE yaml-cpp)

install(TARGETS sched-pipeline
        EXCLUDE_FROM_ALL
        COMPONENT scheduler_applications
        DESTINATION sched)

Snippet from pyproject.toml:

[build-system]
requires = ["py-build-cmake"]
build-backend = "py_build_cmake.build"

[tool.py-build-cmake.module]
directory = "."

[tool.py-build-cmake.sdist]
include = [
    "partition/CMakeLists.txt",
    "partition/sched-pipeline.cpp",
    "partition/yaml_types.h",
    "partition/README.md",
]

[tool.py-build-cmake.cmake]
build_type = "RelWithDebInfo"
source_path = "partition"
build_args = ["-j"]
install_components = ["scheduler_applications"]

[tttapa]

      Traceback (most recent call last):
        File "/Users/cimes/local/fogsys/EdgePipe/venv/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'

It looks like CMake is not installed correctly in the temporary environment that Pip uses to build the package. You'll have to add cmake to the [build-system].requires list in your pyproject.toml file.

Alternatively, you could try uninstalling CMake in your virtual environment (/Users/cimes/local/fogsys/EdgePipe/venv) if you have CMake installed globally.

[cimes-isi]

Interesting - thanks. I have cmake==3.22.2 in my virtualenv and the cmake binary on my PATH is from the virtualenv. Shouldn't cmake be a transitive dependency of py-build-cmake?

[tttapa]

I have cmake==3.22.2 in my virtualenv and the cmake binary on my PATH is from the virtualenv.

Indeed, so when installing, the cmake “binary” used will be the one in your virtualenv. However, it's not really a binary, it's a Python wrapper that imports the cmake module that then invokes the actual binary. Since the build uses its own virtual environment, the import fails, because the cmake module is only installed in your own virtual environment, not the build environment.
I'm not sure why this happens, though, the entrypoints installed by pip should point to the correct Python interpreter, but that doesn't seem to be the case.

Shouldn't cmake be a transitive dependency of py-build-cmake?

I'm kind of on the fence about this: on the one hand it ensures that CMake will always be available, but on the other hand, many systems will have CMake installed globally, in which case it's kind of wasteful to install it through Pip in every virtual environment and for each build.

Instead of making it a hard dependency, I might consider checking whether CMake is available in the get_requires_for_build_wheel hook, and returning it as a dependency if CMake is not in the path or if the one in the path doesn't work.

[cimes-isi]

Something is clearly wrong if it's using the virtualenv's cmake wrapper application which then can't find its cmake module. I'd think if it's building in a new environment, it should just fall back on the (real) system cmake application. Outside my virtualenv:

$ which cmake
/opt/homebrew/bin/cmake

I certainly understand not wanting to add unnecessary dependencies. I'd hope that any workaround you might provide can be implemented reliably, otherwise you may end up playing whack-a-mole. If this is actually a bug in pip, I suggest we open an issue upstream with them. Are you able to reproduce this issue?

[tttapa]

Something is clearly wrong if it's using the virtualenv's cmake wrapper application which then can't find its cmake module. I'd think if it's building in a new environment, it should just fall back on the (real) system cmake application.

It should, but this is not possible, because the bin folder of your virtual environment comes first in the PATH, even when build creates a a sub-environment for you.

You can run python -m build . -C=verbose to see the PATH used when building. For me, it is:

'PATH': '/tmp/build-env-xyzxyzx/bin:/home/pieter/GitHub/py-build-cmake/py-venv/bin:/home/pieter/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin'

As expected, the temporary build environment is first, then your own virtual environment, and finally the system paths.
If your own virtual environment contains a cmake script/binary, it will always pick that one over the global one. To work around this, I would have to patch the PATH manually, which I want to avoid.

I'd hope that any workaround you might provide can be implemented reliably, otherwise you may end up playing whack-a-mole.

I just pushed this change: https://github.com/tttapa/py-build-cmake/commit/5d192449c028b6db5c8eb64f6d6678d3a32c6b94 It conditionally checks whether CMake is available, new enough, and works correctly. If not, it tells the build frontend to install it as a build dependency.

If this is actually a bug in pip, I suggest we open an issue upstream with them. Are you able to reproduce this issue?

I'm not sure, I just checked on my machine (Ubuntu 20.04, Python 3.9, pip 22.0.4), and I get:

$ cat ./py-venv/bin/cmake
#!/home/pieter/GitHub/py-build-cmake/py-venv/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from cmake import cmake
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])

So the shebang is correct and the script should work correctly, regardless of whether the virtual environment is activated or not. I'm not sure if that differs on macOS (I would hope not).

[cimes-isi]

I also see a similar PATH structure during build and my virtualenv cmake wrapper also correctly uses the virtualenv's python3 in the shebang. I can also run it without failure both inside and outside the virtualenv, so the virtualenv's python3 seems to find the cmake package dependency installed in the virtualenv correctly, even when the virtualenv isn't activated. That behavior seems strange to me, but I'm not a Python or virtualenv expert.

So if pip install creates a sub-environment inheriting PATH which has executables that depend on packages installed in the virtualenv, but it can't resolve them, that seems like a bug to me since it correctly found them when the virtualenv isn't activated. My intuition is then that the bug is in whatever package actually configures/manages the sub-environment, which sounds like pip, but I don't know enough about the build process to say with any certainty.

I agree this would be difficult to work around and that you should not try to outsmart the existing PATH.

RE: 5d19244: is cmake 3.20 really the minimum version required? I think a lot of systems won't have that new of a version. Ubuntu 20.04 only has cmake 3.16.3, and 18.04 has 3.10.2.

[tttapa]

So if pip install creates a sub-environment inheriting PATH which has executables that depend on packages installed in the virtualenv, but it can't resolve them, that seems like a bug to me since it correctly found them when the virtualenv isn't activated. My intuition is then that the bug is in whatever package actually configures/manages the sub-environment, which sounds like pip, but I don't know enough about the build process to say with any certainty.

It might actually be a feature, not a bug: pip deliberately disallows access to system packages:

https://github.com/pypa/pip/blob/c247ddce123513198858d4589c5ee198dcc6ba40/src/pip/_internal/build_env.py#L87-L126

I could clear the PYTHONPATH environment variable set by pip before invoking cmake, so that pip's override is not loaded. However, this will have unexpected side effects.

For now, I think the best option is manually checking whether CMake works and installing it if not.

RE: https://github.com/tttapa/py-build-cmake/commit/5d192449c028b6db5c8eb64f6d6678d3a32c6b94: is cmake 3.20 really the minimum version required? I think a lot of systems won't have that new of a version. Ubuntu 20.04 only has cmake 3.16.3, and 18.04 has 3.10.2.

No, older versions should work fine as well. I'll have to test some older versions and select an appropriate value before releasing.

[cimes-isi]

That's interesting, but does pip consider packages in the virtualenv to be "system" packages?

I also feel that clearing PYTHONPATH is not a good idea. I'd hope you wouldn't have to modify anything global like that.

I'm trying to test your recent changes, but pip doesn't seem to want to use the version I installed locally from git into my virtualenv. pip freeze shows -e git+ssh://git@github.com/tttapa/py-build-cmake.git@5d192449c028b6db5c8eb64f6d6678d3a32c6b94#egg=py_build_cmake, but installing my project then with pip install . -vvv seems to still be using 0.0.7, even after purging pip's cache. I feel like I'm missing something silly here - do I need to do something different to force pip install to use the development version?

[tttapa]

That's interesting, but does pip consider packages in the virtualenv to be "system" packages?

I believe so, yes. AFAICT, the code I referred to removes the packages in the virtualenv as well: it removes the paths in get_platlib() and get_purelib(). In my virtual environment, I get:

>>> from pip._internal.locations import get_platlib, get_purelib
>>> get_platlib()
'C:\\Users\\piete\\Documents\\GitHub\\py-build-cmake\\py-venv\\Lib\\site-packages'
>>> get_purelib()
'C:\\Users\\piete\\Documents\\GitHub\\py-build-cmake\\py-venv\\Lib\\site-packages'

do I need to do something different to force pip install to use the development version?

You have to point to the development version in your pyproject.toml build requirements:

[build-system]
requires = ["py-build-cmake @ git+https://github.com/tttapa/py-build-cmake"]
build-backend = "py_build_cmake.build"

[cimes-isi]

That's interesting, but does pip consider packages in the virtualenv to be "system" packages?

I believe so, yes. AFAICT, the code I referred to removes the packages in the virtualenv as well: it removes the paths in get_platlib() and get_purelib(). In my virtual environment, I get:

>>> from pip._internal.locations import get_platlib, get_purelib
>>> get_platlib()
'C:\\Users\\piete\\Documents\\GitHub\\py-build-cmake\\py-venv\\Lib\\site-packages'
>>> get_purelib()
'C:\\Users\\piete\\Documents\\GitHub\\py-build-cmake\\py-venv\\Lib\\site-packages'

Testing this on my system gives similar results, so it looks like you're correct.

do I need to do something different to force pip install to use the development version?

You have to point to the development version in your pyproject.toml build requirements:

[build-system]
requires = ["py-build-cmake @ git+https://github.com/tttapa/py-build-cmake"]
build-backend = "py_build_cmake.build"

I figured it was something stupid-simple like that, thank you. Your changes appear to work without me having to explicitly declare cmake as a build dependency.

---

RE: 5d19244: is cmake 3.20 really the minimum version required? I think a lot of systems won't have that new of a version. Ubuntu 20.04 only has cmake 3.16.3, and 18.04 has 3.10.2.

No, older versions should work fine as well. I'll have to test some older versions and select an appropriate value before releasing.

Related to automatically detecting cmake, I was thinking about version requirements a bit more and realized the a min cmake version may be specified in several places. Without having looked into your codebase, I expect there's at least the CLI compatibility that you need, which I'd expect would allow you to support considerably older versions. However, even if your version requirements are satisfied, the user's top-level project CMakeLists.txt should define their own min version. In more complex setups, a user may even have a tree of cmake projects which each define a potentially different minimum cmake version. The most reliable way to detect if an existing cmake is compatible might be to just run cmake against the source_path specified in pyproject.toml prior to doing the build in the sub-environment, but it could be a fair bit of overhead to run cmake twice. I'm not sure if this is a problem that actually needs to be solved, or if you should just let it fail and/or require the user to specify a min cmake version explicitly in pyproject.toml. Anyway, just thinking out loud.

Cheers.

[tttapa]

I expect there's at least the CLI compatibility that you need, which I'd expect would allow you to support considerably older versions.

I think the main things that affect compatibility are some features of the FindPython3 module.

I'm not sure if this is a problem that actually needs to be solved

Indeed, in my opinion, it's up to the user to make sure a new enough CMake version is available. Trying to find a suitable CMake version for the specific project is beyond the scope of this package. Installing CMake if pip breaks the existing version is reasonable, but I wouldn't go beyond that.

or require the user to specify a min cmake version explicitly in pyproject.toml.

That might be what I'll end up doing.

[cimes-isi]

I expect there's at least the CLI compatibility that you need, which I'd expect would allow you to support considerably older versions.

I think the main things that affect compatibility are some features of the FindPython3 module.

I'd actually argue against that being a concern on your end, I think you should just enforce what is required for py-build-cmake to interact correctly with cmake. For example, my native code doesn't require a Python3 module at all, it's just building more traditional C/C++ without any Python awareness. If a user's CMakeLists uses FindPython3, then any features they require from it may affect their minimum cmake version.

Since finding that module is likely to be a common use case though, there may be value in discussing it in your documentation. Depending on how much detail you provide, it could be a maintenance burden as that cmake module evolves.

That's just my 2 cents though :).

[tttapa]

Thanks, I appreciate your input on this.

I have added a cmake.minimum_version configuration option to pyproject.toml (docs/Config.md#cmake). The default is none (i.e. any version will do, as long as cmake --version exits with status 0).

If you specify this option, e.g. minimum_version = "3.18", and if there is no cmake program in the PATH, if the one in the PATH is broken, or if it is older than the version specified, py-build-cmake will add "cmake>=3.18" to the build requirements, so it will automatically be installed before starting the build.

[cimes-isi]

That appears to work, and is the most straightforward approach I can think of. Users will just have to remember to update the min version spec in both their CMakeLists.txt and in pyproject.toml if/when it changes.

[cimes-isi]

From my perspective, this issue is resolved. Thanks again for your help. Are you planning on making a new release sometime soon? It would be helpful for me so that I can depend on a versioned package in PyPI rather than pointing my build dependency to the github repo. (I don't need it ASAP, but sometime in the near future would be nice.). Cheers.

[tttapa]

I've just created a new release: https://github.com/tttapa/py-build-cmake/releases/tag/0.0.8

[cimes-isi]

Thanks!