ImportError: dlopen on MacOs chip M1

tranchinh87 commented 2 years ago

Hi everyone ! I just migrated to a new M1 MacBook Pro, and i get an error while running a sample on macbook pro with M1 chip.

cd ~/aws-iot-device-sdk-python-v2/samples
# And run the Sample Code
python pubsub.py --endpoint a1nd5owl3rxxxxx-ats.iot.ap-southeast-1.amazonaws.com --root-ca ~/certs/Amazon-root-CA-1.pem --cert ~/certs/device.pem.crt --key ~/certs/private.pem.key

An error has occurred

Traceback (most recent call last):
  File "/Users/xxx/aws-iot-device-sdk-python-v2/samples/pubsub.py", line 5, in <module>
    from awscrt import io, mqtt, auth, http
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/awscrt/io.py", line 11, in <module>
    import _awscrt
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/_awscrt.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_aws_hash_ptr'

My packages installed Python version: 3.10.2 pip list Package Version

awscrt 0.13.3 awsiotsdk 1.9.2 boto3 1.21.15 botocore 1.24.15 jmespath 0.10.0 pip 22.0.4 python-dateutil 2.8.2 s3transfer 0.5.2 setuptools 58.1.0 six 1.16.0 urllib3 1.26.8

jmklix commented 2 years ago

We have yet to test this sdk on M1 Macs. Thanks for letting us know about this ImportError and make sure to add any other errors that you run into below.

tgamushet commented 2 years ago

I have the same issue with awscrt. And awsiot has missing import error but it's installed.

I am trying to run pubsub example too.

M1 MacBook Pro Python 3.9.10 (installed with brew)

pip list

Package         Version
--------------- -------
awscrt          0.13.6
awsiot          0.1.3
boto3           1.21.25
botocore        1.24.26
click           8.0.4
jmespath        1.0.0
pip             22.0.4
python-dateutil 2.8.2
s3transfer      0.5.2
setuptools      60.5.0
six             1.16.0
urllib3         1.26.9

(venv) ➜  aws-iot-client python
Python 3.9.10 (main, Jan 15 2022, 11:40:36) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from awscrt import auth, http, io, mqtt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/awscrt/auth.py", line 8, in <module>
    import _awscrt
ImportError: dlopen(/Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/_awscrt.cpython-39-darwin.so, 2): Symbol not found: _aws_hash_ptr
  Referenced from: /Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/_awscrt.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/_awscrt.cpython-39-darwin.so
>>> from awsiot import mqtt_connection_builder
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'awsiot'

When I install awsiotsdk missing import is fixed but _awscrt comes again with awsiot too.

(venv) ➜  aws-iot-client pip install awsiotsdk 
Collecting awsiotsdk
  Using cached awsiotsdk-1.10.0-py3-none-any.whl (59 kB)
Requirement already satisfied: awscrt==0.13.6 in ./venv/lib/python3.9/site-packages (from awsiotsdk) (0.13.6)
Installing collected packages: awsiotsdk
Successfully installed awsiotsdk-1.10.0
(venv) ➜  aws-iot-client python
Python 3.9.10 (main, Jan 15 2022, 11:40:36) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from awsiot import mqtt_connection_builder
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/awsiot/__init__.py", line 13, in <module>
    from awscrt import mqtt
  File "/Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/awscrt/mqtt.py", line 10, in <module>
    import _awscrt
ImportError: dlopen(/Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/_awscrt.cpython-39-darwin.so, 2): Symbol not found: _aws_hash_ptr
  Referenced from: /Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/_awscrt.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/ym/tga/aws-iot-client/venv/lib/python3.9/site-packages/_awscrt.cpython-39-darwin.so

agodoyAI commented 2 years ago

Following.

tgamushet commented 2 years ago

Manage to get it to work with python-alpine docker image and awsiotsdk==1.10.0 awscrt==0.13.6

awsiotsdk works fine instead of awsiot

graebm commented 2 years ago

We are currently working on fixing this for Apple M1. Skip to the end for a short term workaround.

Background: awsiotsdk depends on awscrt, which contains some C code that needs to be compiled per-platform. We haven't done the work yet to provide precompiled wheels for Apple M1. It also appears that our precompiled wheels for Python 3.9 and 3.10 on macOS are mis-labeled as "universal2" wheels (meaning they should work on either M1 or Intel Macs), but in reality these files only work on Intel right now. The real fix is to provide wheels that work on M1 or Intel. We're currently working on this.

The short term workaround is to compile the code on your own Mac. You can do this by following these steps: 1) Ensure a C compiler is installed on your machine.

In the terminal run: xcode-select --install 2) Ensure CMake is installed on your machine
Either install from a download: https://cmake.org/download/
Or install via homebrew: brew install cmake 3) Re-install awscrt
Uninstall: python3 -m pip uninstall awscrt awsiotsdk
Re-install, but force it to compile locally: python3 -m pip install awsiotsdk --no-binary :all:

tranchinh87 commented 2 years ago

Thank you for your support!

graebm commented 2 years ago

Ok, fixed! Apple Silicon is now an official supported platform for us as of awsiotsdk v1.11.1 Best of luck

aws / aws-iot-device-sdk-python-v2

ImportError: dlopen on MacOs chip M1 #286