search

molinfo-vienna / CDPKit

The Chemical Data Processing Toolkit
https://cdpkit.org
GNU Lesser General Public License v2.1
75 stars 10 forks source link

MacOSX 14 ARM64 build instructions #23

Open pechersky opened 2 days ago

pechersky commented 2 days ago

Hi, thank you for the package and great published papers on how the toolkit works. I'm having issues with arm64 mac wheels as installed from pypi. Doing import CDPL.ConfGen as ConfGen causes an immediate segfault.

So I have been trying to build the wheel on a mac. I've built boost and have python 3.11 installed. My build command is:

python setup.py bdist_wheel -DBOOST_PYTHON_VERSIONS="311" -DPYPI_MANYLINUX_PACKAGE_BUILD=TRUE -DBOOST_INCLUDEDIR="/Users/ec2-user/local/include" -DBOOST_LIBRARYDIR="/users/ec2-user/local/lib" -DBoost_NO_BOOST_CMAKE=TRUE

However, with that, I hit an issue in the InstallExternalRuntimeDependencies step,

 -- Installing external runtime dependencies of /Users/ec2-user/CDPKit/_skbuild/macosx-14.0-arm64-3.11/cmake-install/Libs/Python/CDPL/_cdpl.so...
CMake Error at Libs/Python/CDPL/InstallExternalRuntimeDependencies.cmake:103 (file):
  file Failed to run otool on:

    /install/lib/libpython3.11.dylib
Call Stack (most recent call first):
  Libs/Python/CDPL/InstallExternalRuntimeDependencies.cmake:163 (install_dependencies)
  Libs/Python/CDPL/cmake_install.cmake:62 (include)
  Libs/Python/cmake_install.cmake:42 (include)
  Libs/cmake_install.cmake:47 (include)
  cmake_install.cmake:47 (include)

I tried set(PYTHON_LIBRARIES "") but that leads to 

FAILED: Libs/Python/CDPL/_cdpl.so 
: && /Library/Developer/CommandLineTools/usr/bin/c++ -Wall -Wno-deprecated-declarations -fvisibility=hidden -fvisibility-inlines-hidden -DHAVE_CXX17_FILESYSTEM_SUPPORT -O3 -DNDEBUG -DCDPL_MATH_CHECKS_DISABLE -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX15.0.sdk -mmacosx-version-min=14.0 -bundle -Wl,-headerpad_max_install_names  -o Libs/Python/CDPL/_cdpl.so Libs/Python/CDPL/CMakeFiles/_cdpl.dir/Module.cpp.o Libs/Python/CDPL/CMakeFiles/_cdpl.dir/VersionInfoExport.cpp.o Libs/Python/CDPL/CMakeFiles/_cdpl.dir/ConfigInfoExport.cpp.o Libs/Python/CDPL/CMakeFiles/_cdpl.dir/BuildInfoExport.cpp.o  -Wl,-rpath,/users/ec2-user/local/lib  /users/ec2-user/local/lib/libboost_python311.dylib && :
Undefined symbols for architecture arm64:
  "_PyBool_FromLong", referenced from:
[...]

I saw that Undefined symbols error also separately when I tried to switch from Development to Development.Module.

Thank you for any insight you might provide.

pechersky commented 2 days ago

I put in an install_name_tool call during the install script to move the library name to the true /Library/Frameworks path. And then the find steps proceeded. However, pip installing the resulting wheel still have a warning that it linked in libpython dynamically and would clash with the existing one. And then it segfaults. Running delocate and pip installing that wheel has the same result.

Is there a way to avoid linking in libpython entirely?

seidelt commented 2 days ago

Many thanks for reporting this problem!

Actually, otool shouldn't be executed for the Python runtime library. To prevent this I have added an exclude pattern in the InstallExternalRuntimeDependencies script. However, I have tested the CDPKit build on macOS only with homebrew provided external dependencies. The naming of the runtime library of your Python installation seems to be quite different and the otool exclude pattern thus didn't match. I have now added an extra exclude pattern that should also cover libpython3.11.dylib and thus prevent the failing otool run.

Please pull the latest changes from the github repo and try to build a wheel again.

pechersky commented 1 day ago

Unfortunately, ignoring libpython still results in Python being part of the .dylib. I even built the wheel on a CI machine that has python installed under /Library/Frameworks. Here is the delocate-listdeps --all dist/*.whl, before doing a delocate-wheel:

/Library/Frameworks/Python.framework/Versions/3.11/Python
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
/System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
/Users/runner/local/lib/libboost_iostreams.dylib
/Users/runner/local/lib/libboost_python311.dylib
/opt/homebrew/Cellar/libxau/1.0.11/lib/libXau.6.0.0.dylib
/opt/homebrew/Cellar/libxcb/1.17.0/lib/libxcb.1.1.0.dylib
/opt/homebrew/Cellar/libxdmcp/1.1.5/lib/libXdmcp.6.dylib
/usr/lib/libSystem.B.dylib
/usr/lib/libbz2.1.0.dylib
/usr/lib/libc++.1.dylib
/usr/lib/libexpat.1.dylib
/usr/lib/libz.1.dylib
CDPL/libX11.6.dylib
CDPL/libXau.6.dylib
CDPL/libXdmcp.6.dylib
CDPL/libXext.6.dylib
CDPL/libXrender.1.dylib
CDPL/libcairo.2.dylib
CDPL/libcdpl-base.1.1.dylib
CDPL/libcdpl-biomol.1.1.dylib
CDPL/libcdpl-chem.1.1.dylib
CDPL/libcdpl-confgen.1.1.dylib
CDPL/libcdpl-descr.1.1.dylib
CDPL/libcdpl-forcefield.1.1.dylib
CDPL/libcdpl-grail.1.1.dylib
CDPL/libcdpl-grid.1.1.dylib
CDPL/libcdpl-math.1.1.dylib
CDPL/libcdpl-molprop.1.1.dylib
CDPL/libcdpl-pharm.1.1.dylib
CDPL/libcdpl-shape.1.1.dylib
CDPL/libcdpl-util.1.1.dylib
CDPL/libcdpl-vis.1.1.dylib
CDPL/libfontconfig.1.dylib
CDPL/libfreetype.6.dylib
CDPL/libpixman-1.0.dylib
CDPL/libpng[16](https://github.com/treelinebio/CDPKit/actions/runs/11926807230/job/33241193253#step:8:17).16.dylib
CDPL/libxcb-render.0.dylib
CDPL/libxcb-shm.0.dylib
CDPL/libxcb.1.dylib

and here it is after the delocate-wheel:

$ delocate-listdeps --all wheelhouse/*.whl
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
/System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
/usr/lib/libSystem.B.dylib
/usr/lib/libbz2.1.0.dylib
/usr/lib/libc++.1.dylib
/usr/lib/libexpat.1.dylib
/usr/lib/libz.1.dylib
CDPL/.dylibs/Python
CDPL/.dylibs/libXau.6.0.0.dylib
CDPL/.dylibs/libXdmcp.6.dylib
CDPL/.dylibs/libboost_iostreams.dylib
CDPL/.dylibs/libboost_python311.dylib
CDPL/.dylibs/libxcb.1.1.0.dylib
CDPL/libX11.6.dylib
CDPL/libXau.6.dylib
CDPL/libXdmcp.6.dylib
CDPL/libXext.6.dylib
CDPL/libXrender.1.dylib
CDPL/libcairo.2.dylib
CDPL/libcdpl-base.1.1.dylib
CDPL/libcdpl-biomol.1.1.dylib
CDPL/libcdpl-chem.1.1.dylib
CDPL/libcdpl-confgen.1.1.dylib
CDPL/libcdpl-descr.1.1.dylib
CDPL/libcdpl-forcefield.1.1.dylib
CDPL/libcdpl-grail.1.1.dylib
CDPL/libcdpl-grid.1.1.dylib
CDPL/libcdpl-math.1.1.dylib
CDPL/libcdpl-molprop.1.1.dylib
CDPL/libcdpl-pharm.1.1.dylib
CDPL/libcdpl-shape.1.1.dylib
CDPL/libcdpl-util.1.1.dylib
CDPL/libcdpl-vis.1.1.dylib
CDPL/libfontconfig.1.dylib
CDPL/libfreetype.6.dylib
CDPL/libpixman-1.0.dylib
CDPL/libpng16.16.dylib
CDPL/libxcb-render.0.dylib
CDPL/libxcb-shm.0.dylib
CDPL/libxcb.1.dylib

Taking that wheel, putting it on my EC2, and doing pip install followed by a small import test gives a segfault:

$ python -c "import CDPL.ConfGen as ConfGen"
zsh: segmentation fault  python -c "import CDPL.ConfGen as ConfGen"

Rerunning with debug:

$ python -X faulthandler -c "import CDPL.ConfGen as ConfGen"
Fatal Python error: Segmentation fault

Current thread 0x0000000208190f40 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1233 in create_module
  File "<frozen importlib._bootstrap>", line 573 in module_from_spec
  File "<frozen importlib._bootstrap>", line 676 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/Users/ec2-user/CDPKit/.venv/lib/python3.11/site-packages/CDPL/Math/__init__.py", line 29 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/Users/ec2-user/CDPKit/.venv/lib/python3.11/site-packages/CDPL/Chem/__init__.py", line 28 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/Users/ec2-user/CDPKit/.venv/lib/python3.11/site-packages/CDPL/ForceField/__init__.py", line 27 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/Users/ec2-user/CDPKit/.venv/lib/python3.11/site-packages/CDPL/ConfGen/__init__.py", line 27 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<string>", line 1 in <module>

Extension modules: CDPL._cdpl, CDPL.Base._base (total: 2)
zsh: segmentation fault  python -X faulthandler -c "import CDPL.ConfGen as ConfGen"

I see that in CMakeLists for ManyLinux builds, you set PYTHON_LIBRARIES="". I've been able to build a manylinux wheel for cdpkit fine. Is there a path to doing PYTHON_LIBRARIES="" for Mac wheel builds? Am I not understand some aspect of the libraries that need to be linked?

pechersky commented 1 day ago

Inspection of the _confgen.so:

$ otool -l .venv/lib/python3.11/site-packages/CDPL/ConfGen/_confgen.so
[...]
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 72
         name @loader_path/../libcdpl-confgen.1.1.dylib (offset 24)
   time stamp 2 Thu Jan  1 00:00:02 1970
      current version 1.1.1
compatibility version 1.1.0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 80
         name @loader_path/../.dylibs/libboost_python311.dylib (offset 24)
   time stamp 2 Thu Jan  1 00:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @loader_path/../.dylibs/Python (offset 24)
   time stamp 2 Thu Jan  1 00:00:02 1970
      current version 3.11.0
compatibility version 3.11.0
[...]
$ ls .venv/lib/python3.11/site-packages/CDPL                         
Base                         libX11.6.dylib               libcdpl-math.1.1.dylib
Biomol                       libXau.6.0.0.dylib           libcdpl-molprop.1.1.dylib
Chem                         libXau.6.dylib               libcdpl-pharm.1.1.dylib
ConfGen                      libXdmcp.6.dylib             libcdpl-shape.1.1.dylib
Descr                        libXext.6.dylib              libcdpl-util.1.1.dylib
ForceField                   libXrender.1.dylib           libcdpl-vis.1.1.dylib
GRAIL                        libboost_iostreams.dylib     libfontconfig.1.dylib
Grid                         libboost_python311.dylib     libfreetype.6.dylib
Math                         libcairo.2.dylib             libpixman-1.0.42.2.dylib
MolProp                      libcdpl-base.1.1.dylib       libpixman-1.0.dylib
Pharm                        libcdpl-biomol.1.1.dylib     libpng16.16.dylib
Shape                        libcdpl-chem.1.1.dylib       libxcb-render.0.0.0.dylib
Util                         libcdpl-confgen.1.1.dylib    libxcb-render.0.dylib
Vis                          libcdpl-descr.1.1.dylib      libxcb-shm.0.0.0.dylib
__init__.py                  libcdpl-forcefield.1.1.dylib libxcb-shm.0.dylib
__pycache__                  libcdpl-grail.1.1.dylib      libxcb.1.1.0.dylib
_cdpl.so                     libcdpl-grid.1.1.dylib       libxcb.1.dylib
pechersky commented 1 day ago

Looking at delocate issues, I think there is a suggestion to avoid linking in libpython to begin with: https://github.com/matthew-brett/delocate/issues/25

pechersky commented 1 day ago

For reference, this is the error from the pypi CDPKit:

python -X faulthandler -c "import CDPL.ConfGen as ConfGen"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/ec2-user/CDPKit/.venv/lib/python3.11/site-packages/CDPL/__init__.py", line 37, in <module>
    from ._cdpl import *
ImportError: dlopen(/Users/ec2-user/CDPKit/.venv/lib/python3.11/site-packages/CDPL/_cdpl.so, 0x0002): Library not loaded: /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python
  Referenced from: <3FE3CF2E-6026-3278-BC47-5DF8A72B2444> /Users/ec2-user/CDPKit/.venv/lib/python3.11/site-packages/CDPL/_cdpl.so
  Reason: tried: '/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python' (no such file), '/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python' (no such file)
$ otool -l .venv/lib/python3.11/site-packages/CDPL/_cdpl.so
[...]
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libboost_python312-mt.dylib (offset 24)   # <--- 312??
   time stamp 2 Thu Jan  1 00:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 104
         name /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/Python (offset 24)
   time stamp 2 Thu Jan  1 00:00:02 1970
      current version 3.11.0
compatibility version 3.11.0

Regarding using homebrew python for wheels, the cibuildwheel team suggests to not use brew python for wheel builds (see thread in https://github.com/pypa/cibuildwheel/issues/2034#issuecomment-2398972279).

seidelt commented 23 hours ago

Many thanks for your analytical efforts!

The exclude pattern that I have added for the Python runtime library does not prevent the CDPL modules from linking to it (which is needed for symbol resolution) - that's true! It just makes sure that the library is not added (as other dependencies) to the generated wheel file. The InstallExternalRuntimeDependencies.cmake script takes care of copying the dependencies and also fixes the rpath of the linked libraries. A correct rpath is critical - especially on macOS where a defined absolute or relative path to external libraries needs to be hardcoded into the built binary. Linux is more flexible here and its dynamic linker will search multiple locations for a matching library. Using delocate-wheel is therefore strongly discouraged because it will mess up everything InstallExternalRuntimeDependencies did in a controlled way before.

I will try to reproduce the issues you were observing on my own Mac and hopefully come up with fix in the next days - I will keep you posted!

pechersky commented 21 hours ago

Thank you for your patience and looking into this!

Using delocate-wheel is therefore strongly discouraged

Yet for linux, you run auditwheel repair "${WHEEL_FILE}" which to me seems to be equivalent to delocate-wheel. But I guess your point is that due to the flexibility of the dynamic linker in linux, that's OK.

seidelt commented 18 hours ago

I am running 'auditwheel repair' because it is a requirement for building manylinuxXY compliant Linux wheels accepted by the Python Package Index. Non-standard external dependencies are still installed by the InstallExternalRuntimeDependencies script. 'auditwheel repair' should also be able to do that. However, this never worked for CDPKit wheel builds...