Q: How to enable incremental builds?

[dtromb]

I'm writing an interface to a large-ish library and my compile cycle now takes about 10 minutes.

It is frustrating that using py-build-cmake seems to do everything in a new, fresh environment every time, so that everything I didn't change gets recompiled over, every time.

How can I set things up that that only what needs to be recompiled is (ie. nothing that I haven't changed!)

I tried changing /INCREMENTAL:NO to /INCREMENTAL:YES in the example TOML - but this does seemingly nothing; a new, clean environment is used the next build, and I have to wait the full ten minutes.

[tttapa]

There are two main reasons for the behavior you're seeing:

1. Builds inside of temporary folders created by PyPA build

When you use python -m build, PyPA build will first create a source distribution, and then extract and build that source distribution. This will indeed create a new environment with a CMake build folder in a temporary directory created by PyPA build. This is the desired behavior, because it verifies your sdist.

If you want to cache builds during development, the solution is to use python -m build -w. This builds just the wheel, in a fresh environment, but using the source code from your project folder directly (no sdist), and (re-)using the CMake build folder in project/.py-build-cmake_cache.

Pip's default behavior is close to PyPA build's -w flag, it should also re-use the CMake build folder by default.

If this does not work for you, please check the output of CMake (use the -v flag for pip). It should print -- Build files have been written to: C:/some/folder. You want to verify that it's the same folder every time, and not some temporary folder created by pip or PyPA build.

2. Changing dependencies in virtual environments

If you have dependencies like pybind11 in your [build-system.requires], these are installed by Pip into temporary environments. After every run, they are deleted. The path is also different on each run, so your build system considers these dependencies "dirty", and will rebuild all files that depend on them (e.g. all files that include <pybind11.h>). Possible solutions include:

1. Use a package manager like Conan for these dependencies. Run conan install once, and then re-use libraries like pybind11 across builds. You'll have to tweak the QueryPythonForPybind11 script so it doesn't look for pybind11 in the Python environment.
2. Install the package in some "staging" folder that you add to the CMake search path. Also edit QueryPythonForPybind11.
3. Install the build requirements in your development environment directly, and pass the --no-build-isolation/--no-isolation flags to pip and PyPA build to tell them not to use a temporary virtual environment.

[dtromb]

--no-build-isolation seems perfect (it's worth noting that it seems you must install the package locally once with --no-build-isolation before incremental builds can be done), except after a change to the .cpp nanobind wrapper, py-build-cmake produces the error:

Traceback (most recent call last):
    File "C:\<path>\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 167, in prepare_metadata_for_build_editable
      hook = backend.prepare_metadata_for_build_editable
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: module 'py_build_cmake.build' has no attribute 'prepare_metadata_for_build_editable'

The first install ran perfectly. This only happens after the change. Any ideas how to fix?

[tttapa]

There should be other error messages further up, maybe the CMake build failed because of the change?

[dtromb]

Thanks, this was indeed the case - VC was suppressing its output in a strange way, everything works now! TYVM, appreciate

(perhaps the error message could be improved? - maybe not if there are other conditions where prepare_metadata_for_build_editable might not get set, though... )

[tttapa]

(perhaps the error message could be improved? - maybe not if there are other conditions where prepare_metadata_for_build_editable might not get set, though... )

I'm afraid this AttributeError: module 'py_build_cmake.build' has no attribute 'prepare_metadata_for_build_editable' is raised by Pip, not by py-build-cmake itself, so there's not much I can do about it. It has to do with how Pip deals with the optional prepare_metadata_for_build_editable method in PEP 517.