search

intrepidcs / python_ics

Library for interfacing with Intrepid devices in Python
MIT License
61 stars 29 forks source link

[MacOS arm64]: python-ics 913.16; Incompatible architecture (have 'x86_64', need 'arm64') #168

Closed StephenGemin closed 6 months ago

StephenGemin commented 7 months ago

Problem

It appears that the universal2 wheels uploaded to PyPI do not support arm64 architecture for latest python-ics release, 913.16.

System / Software / Hardware

Stacktrace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/__init__.py", line 11, in <module>
    from ics.ics import *
ImportError: dlopen(/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so, 0x0002): tried: '/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (no such file), '/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/Users/ctest8/.pyenv/versions/3.9.18/envs/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/ctest8/.pyenv/versions/3.9.18/envs/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (no such file), '/Users/ctest8/.pyenv/versions/3.9.18/envs/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Steps to reproduce

Steps below for python 3.9. I've tried 3.9 and 3.11.

# create and activate venv
pip -v install python-ics==913.16
# start interactive python session
import ics

Other Details

I tried building from source following the steps from build_mac.sh. I could not find a way around the below error. This error was present when building from source on both arm64 and x86_64 Mac chips.

In file included from src/dll.cpp:1:
include/dll.h:8:10: fatal error: 'ice/ice.h' file not found
#include <ice/ice.h>
         ^~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1
pierreluctg commented 7 months ago

@drebbe-intrepid, this is because ARCHFLAGS is getting overwritten here https://github.com/intrepidcs/python_ics/blob/master/setup.py#L105

So even if the user/builder sets ARCHFLAGS correctly to -arch x86_64 -arch arm64, only the native arch will be built because it's getting overwritten there!

Two possible solutions:

  1. Change https://github.com/intrepidcs/python_ics/blob/master/setup.py#L105 to append to ARCHFLAGS instead of overwritting and set ARCHFLAGS to -arch x86_64 -arch arm64 in https://github.com/intrepidcs/python_ics/blob/master/build_mac.sh or any other script calling setup.py (like cibuildwheel for example)
  2. Change https://github.com/intrepidcs/python_ics/blob/master/setup.py#L105 to contiue to overwite ARCHFLAGS but also include -arch x86_64 -arch arm64
drebbe-intrepid commented 6 months ago

Problem

It appears that the universal2 wheels uploaded to PyPI do not support arm64 architecture for latest python-ics release, 913.16.

System / Software / Hardware

  • MacOS

    • arm64 architecture (M2 Ultra chip)
    • Sonoma 14.0

  • python-ics 913.16 only

    • I tried python-ics 912.4.post1 and the issue is not reproducible.

Stacktrace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/__init__.py", line 11, in <module>
    from ics.ics import *
ImportError: dlopen(/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so, 0x0002): tried: '/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (no such file), '/Users/ctest8/.pyenv/versions/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/Users/ctest8/.pyenv/versions/3.9.18/envs/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/ctest8/.pyenv/versions/3.9.18/envs/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (no such file), '/Users/ctest8/.pyenv/versions/3.9.18/envs/39_pyics/lib/python3.9/site-packages/ics/ics.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Steps to reproduce

Steps below for python 3.9. I've tried 3.9 and 3.11.

# create and activate venv
pip -v install python-ics==913.16
# start interactive python session
import ics

Other Details

I tried building from source following the steps from build_mac.sh. I could not find a way around the below error. This error was present when building from source on both arm64 and x86_64 Mac chips.

In file included from src/dll.cpp:1:
include/dll.h:8:10: fatal error: 'ice/ice.h' file not found
#include <ice/ice.h>
         ^~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

Thanks for reporting this, I'll look into this ASAP. The ice.h include is a submodule that needs to be checked out now. I'll update the readme to accommodate this change.

drebbe-intrepid commented 6 months ago

@drebbe-intrepid, this is because ARCHFLAGS is getting overwritten here https://github.com/intrepidcs/python_ics/blob/master/setup.py#L105

So even if the user/builder sets ARCHFLAGS correctly to -arch x86_64 -arch arm64, only the native arch will be built because it's getting overwritten there!

Two possible solutions:

  1. Change https://github.com/intrepidcs/python_ics/blob/master/setup.py#L105 to append to ARCHFLAGS instead of overwritting and set ARCHFLAGS to -arch x86_64 -arch arm64 in https://github.com/intrepidcs/python_ics/blob/master/build_mac.sh or any other script calling setup.py (like cibuildwheel for example)
  2. Change https://github.com/intrepidcs/python_ics/blob/master/setup.py#L105 to contiue to overwite ARCHFLAGS but also include -arch x86_64 -arch arm64

Thanks for looking into this. The only change I made was moving the github runner to a newer version (11 to 13). https://github.com/intrepidcs/python_ics/commit/1218156d1f32c0a79069a24773447fa50fb3a301

I'll be looking into this ASAP.

pierreluctg commented 6 months ago

@drebbe-intrepid https://github.com/intrepidcs/python_ics/pull/169 is fixing this issue.

drebbe-intrepid commented 6 months ago

@pierreluctg thank you for fixing this issue! I'll be building a new version with this fix hopefully in the next couple of days. I need to do a release of 914 here also. In the mean time if needed right away the artifacts and wheels are here: https://github.com/intrepidcs/python_ics/actions/runs/7719160743

drebbe-intrepid commented 6 months ago

@pierreluctg @StephenGemin I've just released 914.14 with this fix.