nanobind project unable to static link python library on Windows

[laggykiller]

Original discussion at https://github.com/wjakob/nanobind/issues/262

As adviced, find_python = true was set in py-build-cmake config to help find python in venv, which did help find python in venv (Not sure why this works though).

However...

LINK : fatal error LNK1104: cannot open file 'python38.lib' [D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp38-cp38-win_amd64\_apngasm_python.vcxproj]

See: https://github.com/laggykiller/apngasm-python/actions/runs/5855715610/job/15874048205

Seems like even setting find_python = true does not help find pythonXY.lib? Or venv does not contain pythonXY.lib? Or am I still missing something?

As adviced, I have enabled more verbose debug message in my project. By pure luck, I decided to copy your QueryPythonForNanobind.cmake from develop branch (https://github.com/tttapa/py-build-cmake/blob/develop/examples/nanobind-project/cmake/QueryPythonForNanobind.cmake). Same problem occured on Windows x86 and x64 (Surprisingly not on arm64):

         "D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp38-cp38-win_amd64\ALL_BUILD.vcxproj" (default target) (1) ->
         "D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp38-cp38-win_amd64\_apngasm_python.vcxproj" (default target) (3) ->
         (Link target) -> 
           LINK : fatal error LNK1104: cannot open file 'python38.lib' [D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp38-cp38-win_amd64\_apngasm_python.vcxproj]

See: https://github.com/laggykiller/apngasm-python/actions/runs/5856676134

[laggykiller]

Some insight: From Windows runner that cross-compile arm64:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\arm64\link.exe /ERRORREPORT:QUEUE /OUT:"D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp39-cp39-win_arm64\Release\_apngasm_python.cp39-win_amd64.pyd" /INCREMENTAL:NO /NOLOGO "Release\nanobind-static.lib" apngasm\lib\Release\apngasm.lib "C:\vcpkg\installed\ARM64-windows-static\lib\libpng16.lib" "C:\vcpkg\installed\ARM64-windows-static\lib\zlib.lib" "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\pythonarm64.3.9.10\tools\libs\python39.lib" "C:\vcpkg\installed\arm64-windows-static\lib\zlib.lib" "C:\vcpkg\installed\arm64-windows-static\lib\libpng16.lib" "C:\vcpkg\installed\arm64-windows-static\lib\boost_program_options-vc140-mt.lib" "C:\vcpkg\installed\arm64-windows-static\lib\boost_regex-vc140-mt.lib" "C:\vcpkg\installed\arm64-windows-static\lib\boost_system-vc140-mt.lib" kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:"D:/a/apngasm-python/apngasm-python/.py-build-cmake_cache/cp39-cp39-win_arm64/Release/_apngasm_python.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"D:/a/apngasm-python/apngasm-python/.py-build-cmake_cache/cp39-cp39-win_arm64/Release/_apngasm_python.lib" /MACHINE:ARM64  /machine:ARM64 /DLL _apngasm_python.dir\Release\apngasm_python.obj

See: https://github.com/laggykiller/apngasm-python/actions/runs/5856676134/job/15876928388

Notice that somehow cibuildwheel downloaded python39.lib for us if we are cross-compiling on Windows: C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\pythonarm64.3.9.10\tools\libs\python39.lib

This seems not to occur if we are not cross-compiling.

I am able to recreate the building problem on my local machine by creating venv and build the project. I noticed that venv does not contain static library of python (pythonXY.lib) regardless of which OS.

Seems like nanobind requires static python library when building on Windows (Which is absent in venv), but not on other platforms...?

[laggykiller]

btw (Off topic) In your example CMakeLists.txt (https://github.com/tttapa/py-build-cmake/blob/develop/examples/nanobind-project/CMakeLists.txt), crosscompiling is detected using CMAKE_CROSSCOMPILING, but according to my testing and https://cmake.org/cmake/help/latest/variable/CMAKE_CROSSCOMPILING.html the flag is not set on MacOS cross-compilation, which leads to attempt to create stub on cross-compiled result, causing build to fail. See if that could be improved...?

[laggykiller]

This is interesting...

From https://cmake.org/cmake/help/latest/module/FindPython.html#hints

Python_USE_STATIC_LIBS
If not defined, search for shared libraries and static libraries in that order.

If set to TRUE, search only for static libraries.

If set to FALSE, search only for shared libraries.

Note This hint will be ignored on Windows because static libraries are not available on this platform.

It mentions that static libraries of python is not available on Windows, but how come cmake is trying to find pythonXY.lib, a supposedly non-existing python static library on Windows?

Also I found out there is Python_FIND_VIRTUALENV, but the documentation also mentions It is meaningful only when a virtual environment is active (i.e. the activate script has been evaluated). Not sure if this can help?

[tttapa]

Seems like even setting find_python = true does not help find pythonXY.lib?

It should, and it seems to do so correctly in the simple tests I've run (both locally and in GitHub Actions with CIBW). I'll try to reproduce your problem.

Or venv does not contain pythonXY.lib?

It usually does not, but the actual Python installation backing the venv does, and CMake should still be able to locate it.
If you look at the output of my CIBW run here, you'll see that CMake locates the library in C:/Users/runneradmin/AppData/Local/pypa/cibuildwheel/Cache/nuget-cpython/python.3.8.10/tools/libs/python38.lib. I'm not sure why it fails for your project.

Same problem occured on Windows x86 and x64 (Surprisingly not on arm64):

The difference with ARM64 is that for ARM64, cibuildwheel enables cross-compilation, which explicitly sets the path to pythonXY.lib. (This happens here: https://github.com/pypa/cibuildwheel/blob/ce71f445deee7ac0dabd1ee900d6672370e60478/cibuildwheel/windows.py#L158.)

Seems like nanobind requires static python library when building on Windows (Which is absent in venv), but not on other platforms...?

Indeed, linking a shared library on Linux does not require all undefined symbols to be resolved. Since Python extension modules are loaded by the interpreter (which already links to libpython), the extension modules don't need to link to libpython themselves.
macOS is a similar story, but requires some flags and stubs to silence the linker (see e.g. https://github.com/wjakob/nanobind/blob/master/cmake/darwin-ld-cpython.sym). On Windows, you do need to link against the pythonXY.lib file, which is an import library that simply lists the symbols that are in pythonXY.dll. Even if you don't do anything special in CMake, the linker knows which pythonXY.lib file to link to because of the #pragma comment (lib, pythonXY.lib) in the Python header files (see e.g. https://github.com/python/cpython/blob/6fbaba552a52f93ecbe8be000888afa0b65b967e/PC/pyconfig.h#L315).

btw (Off topic) In your example CMakeLists.txt (https://github.com/tttapa/py-build-cmake/blob/develop/examples/nanobind-project/CMakeLists.txt), crosscompiling is detected using CMAKE_CROSSCOMPILING, but according to my testing and https://cmake.org/cmake/help/latest/variable/CMAKE_CROSSCOMPILING.html the flag is not set on MacOS cross-compilation, which leads to attempt to create stub on cross-compiled result, causing build to fail. See if that could be improved...?

This is now supported in py-build-cmake 0.2.0a2. If the list of architectures (in ARCHFLAGS) does not contain the native architecture, py-build-cmake now automatically enters cross-compilation mode.

how come cmake is trying to find pythonXY.lib, a supposedly non-existing python static library on Windows?

A .lib file can either be a static library or an import library. In this case, it's the import library for pythonXY.dll, so CMake still has to locate it (and should be able to).

---

Please add the following to your pyproject.toml file:

[tool.cibuildwheel]
build-verbosity = 1
environment = { "PY_BUILD_CMAKE_VERBOSE" = "1" }

Also add the following CMake options:

[tool.py-build-cmake.cmake]
build_args = ["-j", "--verbose"]
options = { "CMAKE_FIND_DEBUG_MODE" = "On" }

This should help us debug why CMake can't locate the right library.

For reference, you can look at https://github.com/tttapa/py-build-cmake-example/tree/tttapa-patch-1, which seems to build without any issues.

[tttapa]

Aha, I think I found the issue, you're calling find_nanobind_python_first() (which calls find_package(Python ...)) before the call to project(...). This is not supported, FindPython needs some variables and policies that are set by project(...).

https://github.com/laggykiller/apngasm-python/blob/892076698ff7e383a45aa8160e385ff3fad20cca/CMakeLists.txt#L42

Try moving that line somewhere below the project(...) macro, e.g. to line 125.

[laggykiller]

@tttapa thank you for your advice, now it links and build successfully on Windows!

One small problem though, on MacOS runner that cross-compile arm64:

  CMake Error at cmake/QueryPythonForNanobind.cmake:49 (message):
    Unable to determine extension suffix.  Try manually setting
    PY_BUILD_EXT_SUFFIX.
  Call Stack (most recent call first):
    CMakeLists.txt:121 (find_nanobind_python_first)

That seems to happen after I relocate find_nanobind_python_first() after project(), or after updating py-build-cmake to 0.2.0a2...?

See: https://github.com/laggykiller/apngasm-python/actions/runs/5861479713

Also added more debug option and run again: https://github.com/laggykiller/apngasm-python/actions/runs/5861707991

[tttapa]

Unfortunately, this is a limitation of CMake: when cross-compiling, their FindPython module has some problems, there's essentially no way to get Python_SOABI to the correct value, and they're unwilling to get this fixed.

Since we can't rely on Python_SOABI, I've now explicitly set SETUPTOOLS_EXT_SUFFIX when cross-compiling for macOS (this is also what cibuildwheel does for you when cross-compiling on for Windows ARM64, but for some reason they don't do the same when cross-compiling for macOS ARM64).

py-build-cmake==0.2.0a3 should work (once the CI is finished).

As a side note, the environment you set in your build.yaml overrides the environment you set in pyproject.toml, so I'd recommend adding PY_BUILD_CMAKE_VERBOSE=1 in your build.yaml environment as well, just in case there are any more problems.

[laggykiller]

@tttapa Seems to be working, thanks!

https://github.com/laggykiller/apngasm-python/commit/27b2e8efd60c348f8e3d1a0c0ebe5fc24203bd2a/checks?check_suite_id=15093516262

[tttapa]

@laggykiller It seems that the upload failed because of an invalid Wheel tag when auto-cross-compiling on macOS. This should be fixed in the latest release.
You should be able to just re-run the entire CI run in GitHub Actions (all of them, including the ones that succeeded).

[laggykiller]

Seems to be working, thanks!

I just need to reduce the size of source distribution...

https://github.com/laggykiller/apngasm-python/actions/runs/5872666766/job/15928237414

[laggykiller]

@tttapa Seems like if the macOS target is 11.0 and cross-compiling to arm64, the wheel generated still has incorrect wheel tag of -macosx_11.0_arm64.whl instead of -macosx_11_0_arm64.whl (11.0 instead of 11_0), which cause failure to upload to pypi. This does not occur when I set the macOS target to 10.15 and not cross-compiling.

I am already using the latest version of py-build-cmake

I think the solution might be in src/py_build_cmake/config/quirks.py cross_compile_mac():

cross_arch = get_platform_dashes().split("-")

Should be changed to

cross_arch = get_platform_dashes().replace(".", "_").split("-")

See: https://github.com/tttapa/py-build-cmake/blob/4d838533e6a961625e1c34f22d1fd3cab9b7494b/src/py_build_cmake/config/quirks.py#L164C22-L164C41

Seems like you have written function platform_to_platform_tag() that .replace(".", "_") but it is only called in cross_compile_win() but not cross_compile_mac()

Runner log when _PYTHON_HOST_PLATFORM is macosx-11.0-arm64 (Incorrect tag): https://github.com/laggykiller/apngasm-python/actions/runs/6019714677/job/16329918491

Runner log when _PYTHON_HOST_PLATFORM is macosx-10.15-arm64 (Correct tag): https://github.com/laggykiller/apngasm-python/actions/runs/5874619157/job/15929661777

Alternative solution is setting _PYTHON_HOST_PLATFORM to macosx-11_0-arm64 instead of macosx-11.0-arm64 (https://github.com/laggykiller/apngasm-python/actions/runs/6020714873)? This is against the advice from many sources, including your documentation: https://tttapa.github.io/py-build-cmake/FAQ.html#how-to-build-my-package-for-many-python-versions-operating-systems-and-architectures

[tttapa]

This issue was fixed on the develop branch in 6ebb2960becd2e7402047bbf0c514867bad73059, but not yet merged into the rework-0.2.0 branch. I've merged it now.

I'm currently working on a significant refactor, and the alpha releases that come out of that effort should not be considered stable by any means, they are meant for testing purposes only. When the refactor is complete and I've thoroughly tested everything, I'll make a stable 0.2.0 release.

In the meantime, if you need certain improvements that are not yet available in the latest stable release, I'd recommend pinning a specific pre-release version in your requirements, (i.e. using py-build-cmake==0.2.0a7, not >=0.2.0a7).