Installing and locating cmake-compiled executables

[cimes-isi]

As I mentioned in #3, I'm building an executable---rather than direct Python bindings---which I run as a subprocess. The question is then where and how (with py-build-cmake) to install the executable in a standard way.

I believe there are at least the following requirements:

1. The installed executable must be discoverable by at least the Python module that created it. More broadly, it might be discoverable by any Python code in the environment, even if that requires it to know what module owns the executable.
2. The executable path must be resolvable by the OS so that it can be executed by subprocess.run(...). This would seem to require that it must be installed on the filesystem, i.e., not in a zipped package. (For example, it may not be sufficient to treat the executable as a resource that can be located, e.g., with importlib.resources.)

The easiest thing is for the installed executable to be on PATH so that no additional searching is required. This is also perhaps the most general result we can achieve.

py-build-cmake appears to use the site-packages directory as the cmake install prefix (at least in my virtualenv), but I don't believe this or any subdirectories can be assumed (or even expected) to be on PATH. Can/should we be able to install to the bin/ directory of a virtualenv? I don't know enough about Python packaging and distribution to understand if that generalizes or if there's a standard that achieves a similar result (like an entry point, even if that requires a level of indirection?).

Thanks!

[tttapa]

py-build-cmake appears to use the site-packages directory as the cmake install prefix

This is not specific to py-build-cmake, it's because of how wheel packages are structured. AFAIK, you cannot directly package files to be installed in folders other than site-packages/package-name.

The officially supported approach would be to use [project.scripts] (similar to setuptools entry points): https://flit.pypa.io/en/latest/pyproject_toml.html#scripts-section

This is what the Python CMake package uses:

• https://github.com/scikit-build/cmake-python-distributions/blob/94ac8902fb2e7df773616506cafe6291e47f9dc8/setup.py#L46-L50
• https://github.com/scikit-build/cmake-python-distributions/blob/94ac8902fb2e7df773616506cafe6291e47f9dc8/src/cmake/__init__.py

but I don't believe this or any subdirectories can be assumed (or even expected) to be on PATH

Indeed.

Can/should we be able to install to the bin/ directory of a virtualenv?

I don't think so. Pip does create wrapper scripts in bin/ when installing packages that contain scripts/entry points.

[cimes-isi]

py-build-cmake appears to use the site-packages directory as the cmake install prefix

This is not specific to py-build-cmake, it's because of how wheel packages are structured. AFAIK, you cannot directly package files to be installed in folders other than site-packages/package-name.

To clarify, I meant the site-packages directory itself, not site-packages/package-name. I would've been less surprised if it had been the latter. As it is now, I can install to anywhere in site-packages, potentially even overwriting other package files. I expect that would be problematic.

The officially supported approach would be to use [project.scripts] (similar to setuptools entry points): https://flit.pypa.io/en/latest/pyproject_toml.html#scripts-section

Thanks. I'll explore this in more detail next week and post back here. If it seems like the right solution, it might be worth documenting.

[tttapa]

To clarify, I meant the site-packages directory itself, not site-packages/package-name. I would've been less surprised if it had been the latter. As it is now, I can install to anywhere in site-packages, potentially even overwriting other package files. I expect that would be problematic.

You're correct. I hadn't actually considered writing to site-packages directly, and I'm not sure what the PEP/PyPA guidelines have to say about that.

If it seems like the right solution, it might be worth documenting.

I did some more research, and it seems you can install binaries to {distribution}-{version}.data/scripts, and pip will install them to a folder in the path (e.g. $VIRTUAL_ENV/bin).

I've added an example here: examples/minimal-program

[cimes-isi]

Thanks, that seems to work with pip in my virtualenv. I generalized it a bit so the cmake project still works as expected outside of py-build-cmake, e.g.:

if(DEFINED PY_BUILD_CMAKE_PACKAGE_NAME AND DEFINED PY_BUILD_CMAKE_PACKAGE_VERSION)
  message(STATUS "Checking for py-build-cmake environment - found")
  message(STATUS "  Using PEP 427-compatible install paths")
  set(CMAKE_INSTALL_BINDIR ${PY_BUILD_CMAKE_PACKAGE_NAME}-${PY_BUILD_CMAKE_PACKAGE_VERSION}.data/scripts)
else()
  message(STATUS "Checking for py-build-cmake environment - not found")
  message(STATUS "  Using default install paths")
endif()

install(TARGETS minimal_program
        COMPONENT python_binaries
        RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})

I'm unclear if there's any benefit to using EXCLUDE_FROM_ALL in your example, but I don't see any ill effect by removing it. Do you ever issue an install command for the all target (w/out specifying the component)? As an aside, this flag forces a min cmake version of 3.6 (see install docs).

It would be nice if standard install paths could be directly set by py-build-cmake so that the cmake projects don't have to have awareness of py-build-cmake (via PY_BUILD_CMAKE_* variables), but I'm not sure there's a clean way to do so (for example, some googling suggests that if the project uses GNUInstallDirs, then overriding even just a single value like CMAKE_INSTALL_BINDIR may block GNUInstallDirs from setting all its other variables) or if doing so might unnecessarily constrain cmake project behaviors from the user's perspective. I think what you have now is therefore appropriate, and if the future allows better generalization, then it can be addressed at that time.

[tttapa]

Do you ever issue an install command for the all target (w/out specifying the component)?

Yes, this is quite common, for many projects you simply use cmake; make; make install, and you usually don't want to include any Python-specific files in that case.

It would be nice if standard install paths could be directly set by py-build-cmake so that the cmake projects don't have to have awareness of py-build-cmake

I'm not sure, I think it's best to be explicit about this in the CMake script. Having py-build-cmake override standard options like CMAKE_INSTALL_BINDIR is risky, and adding non-standard variables (e.g. PY_BUILD_CMAKE_BINDIR) would only result in more “unused variables” warnings.

[cimes-isi]

As you say, there are of course scenarios that might need Python awareness w.r.t. install directories, e.g., if they conditionally build Python bindings for native code that is also used more broadly. In that case, they may need to make a distinction on where to install different libraries and executables, and thus it makes sense to rely on PY_BUILD_CMAKE_* variables like you introduced.

Solely configuring the standard variables might be useful for code that either (1) is entirely Python-agnostic (e.g., in my example where I just build an executable without Python awareness but which is used by Python code) or (2) is only ever built in a Python environment, e.g., to accelerate Python codebases with natively compiled code. Then the cmake-managed builds could still be agnostic to the higher-level tool (py-build-cmake in this case) by only using CMake-documented parameters. It would then be easy for users to swap those higher-level tools without having to change their CMakeLists.txt.

I agree that introducing non-standard install variables is not ideal - I was referring to the ones you already introduced (sorry that wasn't clear). If you were to support configuring the standard install directories, I'd expect you'd want it to be configurable. In any case, I haven't exhaustively thought it through and am not requesting that you implement it now. I think the approach you provided is sufficient for my case at this time. Thanks!

[tttapa]

In cases where you're packaging a Python-unaware CMake project, you'll have to include a pyproject.toml file anyway, so I think it's best to then also add a wrapper CMakeLists.txt file where you just set(CMAKE_INSTALL_BINDIR ...) and add_subdirectory(the_actual_project).

If you have a Python-aware project that can also be used as standalone C++ project (e.g. a C++ library and tools with Python bindings), I think it makes sense to have different installation components with different installation paths (e.g. bin, shlib, dev components with the default GNUInstallDirs paths, and python_modules or python_package components with the Wheel-specific paths).

Trying to perform a generic install into a Wheel package by simply (and automatically) changing the installation directories is bound to fail for any nontrivial project, so it's best to be explicit about it.

[cimes-isi]

I see 0.0.9 now adds a distinction between PY_BUILD_CMAKE_PACKAGE_NAME and PY_BUILD_CMAKE_MODULE_NAME, so I'll use the latter. I think we can mark this is as resolved. Thanks again.