Linker undefined symbol errors using build-wheel to build highspy

[BenAokiSherwoodAPL]

Hi,

I am trying to build a wheel for highspy, and when I run ./build-wheel.py --python 3.11 --abi x86_64 highspy, cmake is able to build all the objects but then the build fails on the linking step, throwing linker errors such as ld: error: undefined symbol: PyErr_Fetch. The output log is here: build_highspy_log.txt.

My packages/highspy directory contains only meta.yaml with the following contents:

package: 
 name: highspy
  version: "1.7.1.dev1"

I have no experience building packages, so I'm not sure how to even start tackling these errors. Any advice is much appreciated, and please let me know if further context is needed. Thanks!

[mhsmith]

The errors are because it's not linking against libpython. In the cmake-example recipe, we fix this by patching pybind11's CMakeLists file, but that probably wouldn't work here since the package installs pybind11 using pyproject.toml rather than using a submodule.

The easiest way to work around this is probably to add an-lpython3.11 argument on the linker command line. build-wheel.py tries to do this using LDFLAGS, but I guess CMake doesn't pay attention to that.

However, there's probably a CMake variable that would do it. If you can work out which one, you can add it to the Python section of generate_cmake_toolchain in build-wheel.py.

[BenAokiSherwoodAPL]

Thank you for the advice. After a bunch more digging around, I'm still unable to get the build working. I first tried adding target_link_libraries(TARGET -lpython3.11) for various TARGETs including _core and highs in the Python section of generate_cmake_toolchain as you suggested, but this always resulted in the following error:

./build-wheel.py --python 3.11 --abi x86_64 highspy --no-unpack
build-wheel: cd /home/aokisbg1-a/chaquopy/server/pypi/packages/highspy/build/1.7.1.dev1
build-wheel: Reusing existing build directory due to --no-unpack
build-wheel: abi=x86_64; api_level=21; prefix=/home/aokisbg1-a/chaquopy/server/pypi/packages/highspy/build/1.7.1.dev1/cp311-cp311-android_21_x86_64/requirements/chaquopy; . /home/aokisbg1-a/chaquopy/server/pypi/../../target/build-common.sh; export
build-wheel: /home/aokisbg1-a/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aokisbg1-a/chaquopy/server/pypi/packages/highspy/build/1.7.1.dev1/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib/libpthread.a
build-wheel: /home/aokisbg1-a/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aokisbg1-a/chaquopy/server/pypi/packages/highspy/build/1.7.1.dev1/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib/librt.a
build-wheel: cd /home/aokisbg1-a/chaquopy/server/pypi/packages/highspy/build/1.7.1.dev1/cp311-cp311-android_21_x86_64/src
build-wheel: Building with PEP 517
*** scikit-build-core 0.9.2 using CMake 3.29.2 (wheel)
*** Configuring CMake...
2024-04-26 11:58:06,856 - scikit_build_core - WARNING - libdir/ldlibrary: /home/aokisbg1-a/miniconda3/envs/build-wheel/lib/libpython3.11.a is not a real file!
2024-04-26 11:58:06,856 - scikit_build_core - WARNING - Can't find a Python library, got libdir=/home/aokisbg1-a/miniconda3/envs/build-wheel/lib, ldlibrary=libpython3.11.a, multiarch=x86_64-linux-gnu, masd=None
loading initial cache file /tmp/tmpcgrd4h5_/build/CMakeInit.txt
CMake Error at /home/aokisbg1-a/chaquopy/server/pypi/packages/highspy/build/1.7.1.dev1/cp311-cp311-android_21_x86_64/chaquopy.toolchain.cmake:21 (target_link_libraries):
  Cannot specify link libraries for target "_core" which is not built by this
  project.
Call Stack (most recent call first):
  /home/aokisbg1-a/chaquopy/server/pypi/packages/highspy/build/1.7.1.dev1/cp311-cp311-android_21_x86_64/env/lib/python3.11/site-packages/cmake/data/share/cmake-3.29/Modules/CMakeDetermineSystem.cmake:146 (include)
  CMakeLists.txt:23 (project)

-- Configuring incomplete, errors occurred!

*** CMake configuration failed
build-wheel: Error: Backend subprocess exited when trying to invoke build_wheel

After that, I started looking into the build configuration specified in highspy's pyproject.toml. The one lead I found is that, after build-wheel.pycompletes its pre-processing and passes off the building to the backend subprocess, scikit-build, I get the following warnings:

build-wheel: Building with PEP 517
*** scikit-build-core 0.9.2 using CMake 3.29.2 (wheel)
*** Configuring CMake...
2024-04-26 12:05:41,713 - scikit_build_core - WARNING - libdir/ldlibrary: /home/aokisbg1-a/miniconda3/envs/build-wheel/lib/libpython3.11.a is not a real file!
2024-04-26 12:05:41,713 - scikit_build_core - WARNING - Can't find a Python library, got libdir=/home/aokisbg1-a/miniconda3/envs/build-wheel/lib, ldlibrary=libpython3.11.a, multiarch=x86_64-linux-gnu, masd=None
loading initial cache file /tmp/tmpe1cdt795/build/CMakeInit.txt
...

This happens right at the start of scikit-build's Cmake configuration step; after these warnings, I can see that Cmake is working through CMakeLists.txt in the src directory of the unpacked build directory. Beyond that, I'm at a loss. The actual error (see log attached to my previous message) occurs during the linking phase in Ninja, and I'm not sure how the build outputs of the CMake stage affect the Ninja stage.

[mhsmith]

Cannot specify link libraries for target "_core" which is not built by this project.

In that case, you should find the CMakeLists file which defines that target, and make the edit there instead.