Build fails on macOS M1 Apple Silicon in Eclipse, while it works on CLI (IEP-545)

georgik commented 3 years ago

Build of hello_world fails of macOS M1 with following error:

ImportError: dlopen(/Users/georgik/.espressif/python_env/idf4.4_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so, 2): no suitable image found.  Did find:
    /Users/georgik/.espressif/python_env/idf4.4_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: mach-o, but wrong architecture
    /Users/georgik/.espressif/python_env/idf4.4_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: mach-o, but wrong architecture
ninja: build stopped: subcommand failed.

The build works without problem when build from terminal.

To Reproduce Steps to reproduce the behavior:

Create hello_world project and build it.
The build fail in later stage when it's invoking Python which requires cryptography

Expected behavior The build pass

Environment Information: Eclipse 2021-06 x86_64

ESP-IDF Tools Information: ESP-IDF 4.4

Problem analysis The problem is caused by invoking Python in x86_64 mode instead of arm64. Python wheels for M1 is available in arm64 format, the Python executed in x86_64 mode will try to seek for .so which is for x86_64.

Workaround The workaround is to launch Python which has only M1 version, e.g. one from Homebrew, not one from Apple. This Python works:

/opt/homebrew/bin//python3: Mach-O 64-bit executable arm64

This Python won't work:

% file /usr/bin/python3
/usr/bin/python3: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/python3 (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/bin/python3 (for architecture arm64e): Mach-O 64-bit executable arm64e

Workaround steps:

remove ~/.espressif/python_env/idf*
explicitly set priotity to PATH to Homebrew Python: export PATH=/opt/homebrew/bin/:$PATH
double check the Python: file `which python3`- only one arch should be displayed
run install.sh
update Eclipse configuration to point to newly created virtualenv

Possible solution: Force launching Python in arm64 for mode. There should be CLI option for macOS M1 to do it

kolipakakondal commented 3 years ago

Hi @georgik

Do we need this PR to work with the Apple M1 or was it working fine with arm64 based eclipse and java 17?

There are two things we did in the PR

Passing arch -arm64 argument to python
Setting correct python version to the java process runner >>> this is anyway required

Builds Java 17 for macOS arm64 https://download.oracle.com/java/17/latest/jdk-17_macos-aarch64_bin.dmg

Eclipse CPP arm64 build for macOS - this is a prerelease version eclipse-cpp-2021-12-M1-macosx.cocoa.aarch64

expresspotato commented 2 years ago

Any luck with this? Still experiencing the same issue here on 2.4.2... idf.py builds the project fine, but fails in eclipse due to the architecture mismatch with python required libs.

kolipakakondal commented 2 years ago

Hi @expresspotato

Could you confirm whether you are working with AArch64 builds from eclipse?

You could download it directly from here https://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/2022-03/R/eclipse-cpp-2022-03-R-macosx-cocoa-aarch64.dmg

expresspotato commented 2 years ago

Yup...

kevin@Kevins-MacBook-Pro MacOS % pwd
/Applications/Eclipse.app/Contents/MacOS
kevin@Kevins-MacBook-Pro MacOS % file eclipse 
eclipse: Mach-O 64-bit executable arm64

Version: 2022-03 (4.23.0)

Steps to reproduce:

Download Eclipse IDE for C/C++ AArach64
Install the plugin according to the directions
Load up example project WiFi -> Roaming
Build with ESP-IDF Application and esp32 target

FAILED: esp-idf/mbedtls/x509_crt_bundle 
cd /Users/kevin/workspace/roaming/build/esp-idf/mbedtls && python /Users/kevin/esp/esp-idf/components/mbedtls/esp_crt_bundle/gen_crt_bundle.py --input /Users/kevin/esp/esp-idf/components/mbedtls/esp_crt_bundle/cacrt_all.pem -q
Traceback (most recent call last):
  File "/Users/kevin/esp/esp-idf/components/mbedtls/esp_crt_bundle/gen_crt_bundle.py", line 224, in <module>
    main()
  File "/Users/kevin/esp/esp-idf/components/mbedtls/esp_crt_bundle/gen_crt_bundle.py", line 208, in main
    bundle.add_from_file(path)
  File "/Users/kevin/esp/esp-idf/components/mbedtls/esp_crt_bundle/gen_crt_bundle.py", line 86, in add_from_file
    self.add_from_pem(crt_str)
  File "/Users/kevin/esp/esp-idf/components/mbedtls/esp_crt_bundle/gen_crt_bundle.py", line 116, in add_from_pem
    self.certificates.append(x509.load_pem_x509_certificate(crt.encode(), default_backend()))
  File "/Users/kevin/.espressif/python_env/idf4.3_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/backends/__init__.py", line 14, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/kevin/.espressif/python_env/idf4.3_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 6, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/kevin/.espressif/python_env/idf4.3_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 113, in <module>
    from cryptography.hazmat.bindings.openssl import binding
  File "/Users/kevin/.espressif/python_env/idf4.3_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: dlopen(/Users/kevin/.espressif/python_env/idf4.3_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so, 0x0002): tried: '/Users/kevin/.espressif/python_env/idf4.3_py3.8_env/lib/python3.8/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
ninja: build stopped: subcommand failed.

kolipakakondal commented 2 years ago

Could you also make sure Java is arm64 based build? This will make sure all the child processes(eclipse, python, etc) also considered as arm64 https://download.oracle.com/java/17/latest/jdk-17_macos-aarch64_bin.dmg

In meantime, I will check and get back to you. Maybe we need to pass arch -arm64 for python as suggested by @georgik .

expresspotato commented 2 years ago

Could you also make sure Java is arm64 based build? This will make sure all the child processes(eclipse, python, etc) also considered as arm64 https://download.oracle.com/java/17/latest/jdk-17_macos-aarch64_bin.dmg

In meantime, I will check and get back to you. Maybe we need to pass arch -arm64 for python as suggested by @georgik .

Thanks for the suggestion, but it doesn't work here on M1.

espressif / idf-eclipse-plugin

Build fails on macOS M1 Apple Silicon in Eclipse, while it works on CLI (IEP-545) #376