Cryptography 36.0.2 (and rust support)

[emanuele-f]

Here are some notes I took when building the cryptography 36.0.2 python module in Chaquopy, which was needed to run mitmproxy 8.0.0. I hope this will be useful to integrate it in the officially supported packages.

Overview

mitmproxy 8.0.0 depends on the new cryptography 36.x module, for which rust is now mandatory. cryptography uses setuptools_rust to build native shared objects and pyo3 to invoke them from python.

Here the relevant part of the dependency tree:

mitmproxy
  cryptography
    setuptools_rust
      rust
    pyo3
      python

Other than rust setup for cross-compilation, the problematic part is that cryptography uses pyo3 v0.15.1, which requires a python interpreter to be cross-compiled for the target architecture (and it looks for _sysconfigdata* in the python lib folder). In theory, both pyo3 and cryptography support building abi3 modules, which work regardless of the specific python version and could be built without an existing python interpreter. However, this possibility is only added in pyo3 0.16.4, which is not currently supported by cryptography. See https://github.com/PyO3/pyo3/discussions/2310 for more details.

So, in order to build the cryptography module, we currently need:

• Install rust and configure it for the cross compilation
• Cross compile python

With some tricks, it's possible to compile only the core of python without the need to also cross-compile its dependencies. In fact, the pyo3 shared modules of cryptography will be linked against the soname used by python library shipped with chaquopy.

Preparing the build environment

The following instructions are meant for building on an archlinux host without docker. Install requirements (most requirements not listed here, see target/Dockerfile)

pacman -S patchelf

Copy the Android toolchains used by chaquopy:

docker build -t chaquopy-base -f base.dockerfile .
docker build -t chaquopy-target target

# replace 62b7d2f98872 with the chaquopy-target container ID
cd target
docker cp 62b7d2f98872:/root/target/toolchains toolchains

To avoid polluting the build machine, use an overlayfs to install all the stuff:

cd /home/emanuele/src/build-wheel
mkdir build
cd build
rm -rf sysroot
mkdir -p sysroot overlay workdir

# NOTE: to refresh the lowerdir when mounted, run "sudo mount -o remount overlay"
sudo mount -t overlay overlay -o lowerdir=/,upperdir=./sysroot,workdir=./workdir ./overlay
sudo arch-chroot ./overlay

Build and install the exact python version required by Chaquopy:

version=3.8.7

cd /home/emanuele/src/build-wheel/build
wget https://www.python.org/ftp/python/$version/Python-$version.tgz

tar -xf Python-$version.tgz
cd Python-$version
./configure --prefix=/usr
make -j $(nproc)
make install

# verify
python3.8 --version

Install rust and libraries required for cross-compilation in rust

pacman -Rs rustup rust
curl https://sh.rustup.rs -sSf | sh -s -- -y
source "$HOME/.cargo/env"

# required for cross-compilation
rustup target add aarch64-linux-android
rustup target add arm-linux-androideabi
rustup target add i686-linux-android
rustup target add x86_64-linux-android

# Verify installed cross-compilation libraries
rustc --print target-list | grep android

# install requirements, they will be needed to build cryptography
cd /home/emanuele/src/build-wheel/server/pypi
pip3.8 install -r requirements.txt

Building for an ABI

These steps must be performed for each Android ABI.

First select the target ABI:

# only pick the target ABI of choice
export ARCH="armeabi-v7a" CPU="arm"     TOOL_PREFIX="arm-linux-androideabi" TOOL_CFLAGS="-march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb" TOOL_LDFLAGS="-march=armv7-a -Wl,--fix-cortex-a8"
export ARCH="arm64-v8a"   CPU="aarch64" TOOL_PREFIX="aarch64-linux-android" TOOL_CFLAGS="" TOOL_LDFLAGS=""
export ARCH="x86"         CPU="x86"     TOOL_PREFIX="i686-linux-android"    TOOL_CFLAGS="" TOOL_LDFLAGS=""
export ARCH="x86_64"      CPU="x86_64"  TOOL_PREFIX="x86_64-linux-android"  TOOL_CFLAGS="" TOOL_LDFLAGS=""

Setup the environment for cross compilation:

version=3.8.7
cd /home/emanuele/src/build-wheel/build

# cleanup and setup
rm -rf $ARCH/sysroot/usr $ARCH/Python-$version
mkdir -p $ARCH/sysroot/usr/lib $ARCH/sysroot/usr/include
tar -C $ARCH -xf Python-$version.tgz

TOOLCHAIN=`readlink -f ../target/toolchains`
SYSROOT=`readlink -f $ARCH/sysroot`

# copy headers from the toolchain to the sysroot
cp -r $TOOLCHAIN/$ARCH/sysroot/usr/* $SYSROOT/usr

# (OPTIONAL) extend sysroot with python deps, not actually needed by pyo3
#cp -r deps/OpenSSL-for-Android-Prebuilt/openssl-1.1.1k-clang/include/* $ARCH/sysroot/usr/include
#cp -r deps/libcrypt_0.2-5_aarch64/usr/* $ARCH/sysroot/usr
#...

# Needed to cross compile python (some may not be needed)
export CC="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-gcc"
export CXX="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-g++"
export AR="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-ar"
export AS="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-as"
export LD="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-ld"
export LDSHARED="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-gcc -shared"
export RANLIB="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-ranlib"
export STRIP="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-strip --strip-unneeded"
export NM="$TOOLCHAIN/$ARCH/bin/$TOOL_PREFIX-nm"
export READELF="$TOOLCHAIN/$ARCH/$TOOL_PREFIX/bin/readelf"
export CROSS_COMPILE_TARGET=yes
export CFLAGS="-fPIC -DANDROID --sysroot=$SYSROOT $TOOL_CFLAGS"
export CXXFLAGS="$CFLAGS"

# Flags taken from build-wheel.py
export LDFLAGS="--sysroot=$SYSROOT -Wl,--no-undefined -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libunwind.a $TOOL_LDFLAGS"

Cross compile python

cd $ARCH/Python-$version
./configure --build=x86_64-unknown-linux-gnu --host=$CPU-linux-android --enable-shared --prefix=`readlink -f ../sysroot/usr` \
    ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no ac_cv_have_long_long_format=yes ac_cv_buggy_getaddrinfo=no

# patch to support Android API 19
# chaquopy targets API 19 on armv7a/x86, which is not fully supported by python https://bugs.python.org/issue36162
# the API level is specified via the "--target=" arg inside the $TOOL_PREFIX-gcc script
sed -i 's/^#define HAVE_SENDFILE 1$/\/* #undef HAVE_SENDFILE *\//g' pyconfig.h
sed -i 's/^#define HAVE_TRUNCATE 1$/\/* #undef HAVE_TRUNCATE *\//g' pyconfig.h
sed -i 's/^#define HAVE_WCSFTIME 1$/\/* #undef HAVE_WCSFTIME *\//g' pyconfig.h

# build (ignore errors, they occur because of missing dependencies, but we are only interested in libpython and related _sysconfigdata)
make -j $(nproc)
make install

# Patch SONAME to correspond to the chaquopy "libpython3.8.so"
# check with x86_64-linux-android-readelf -Wa $SYSROOT/usr/lib/libpython3.8.so | grep SONAME
mv $SYSROOT/usr/lib/libpython3.8.so{.1.0,}
patchelf --set-soname libpython3.8.so $SYSROOT/usr/lib/libpython3.8.so

Finally, build cryptography (assume rust_cross_compile.patch is applied (see below), which sets PYO3_CROSS_LIB_DIR):

cd /home/emanuele/src/build-wheel/server/pypi

python3.8 ./build-wheel.py --toolchain ../../target/toolchains/$ARCH cryptography

If build successful, you can retrieve the built wheel from outside the overlayfs in the build-wheel/sysroot folder.

build-wheel patches

rust_cross_compile.patch:

--- src-original/setup.py
+++ src/setup.py
@@ -10,6 +10,16 @@ import sys

 from setuptools import setup

+# https://doc.rust-lang.org/rustc/codegen-options/index.html
+os.environ["RUSTFLAGS"] = f"-C linker={os.environ['CC']}"
+os.environ["CARGO_BUILD_TARGET"] = os.environ['CHAQUOPY_TRIPLET']
+
+# https://pyo3.rs/v0.15.2/building_and_distribution.html
+os.environ["PYO3_PYTHON"] = f"python{os.environ['CHAQUOPY_PYTHON']}"
+os.environ["PYO3_CROSS_PYTHON_VERSION"] = os.environ['CHAQUOPY_PYTHON']
+os.environ["PYO3_CROSS_LIB_DIR"] = f"{os.environ['RECIPE_DIR']}/../../../../build/{os.environ['CHAQUOPY_ABI']}/sysroot/usr/lib"
+#print(os.environ)
+
 try:
     from setuptools_rust import RustExtension
 except ImportError:

cryptography meta.yaml:

package:
  name: cryptography
  version: 36.0.2

requirements:
  build:
    - cffi 1.13.2
    - setuptools-rust 1.2.0
  host:
    - openssl

Add rust to the docker build script

# Install rust and rustup
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y

# Install crates needed for cross compilation (note you need to re-run all the build commands)
# https://rust-lang.github.io/rustup/cross-compilation.html
Run source $HOME/.cargo/env && \
    rustup target add x86_64-linux-android && \
    rustup target add i686-linux-android && \
    rustup target add aarch64-linux-android && \
    rustup target add armv7-linux-androideabi

[mhsmith]

Thanks very much. I'm currently working on updating Chaquopy to Python 3.10 (#624), so I'll have a look at updating Cryptography after that. The cleanest solution would probably be to take the existing Rust support from the tokenizers package and merge that into the main build-wheel tool, along with any necessary additions from this issue.

If anyone else needs a new version of this package, please click the thumbs up button above, and post a comment explaining why you need it.

[newfix]

First of all, I would like to thank you for creating and sharing Chaquopy. I am truly impressed how easy it is to run Python code on Android using this SDK. I am currently using Chaquopy (via BeeWare / Briefcase) to develop an Android app in Python that needs to encrypt and decrypt data using cryptographic primitives which are only available in recent versions of the Cryptography package. Therefore, I am interested in getting new versions of the Cryptography package (preferably the latest version) to work with Chaquopy. So far, all my attempts to install version 39 have been without success. Please let me know if there is an (easy) way to make it work or if a new version of Cryptography for Chaquopy is to be expected soon.

[mhsmith]

Would version 36.0.2 be new enough for you? If so, you can try using the wheels built by @emanuele-f at https://github.com/emanuele-f/chaquopy-wheels.

If you need a newer version than that, please let me know exactly which features you need it for.

[newfix]

Unfortunately, version 36.0.2 is not suitable for me, given that I have to use AESSIV which was introduced with version 37 (see https://cryptography.io/en/latest/hazmat/primitives/aead/#cryptography.hazmat.primitives.ciphers.aead.AESSIV). The performance improvement of ChaCha20Poly1305 introduced with version 39 would be a bonus, but is not strictly necessary.

[emanuele-f]

I will probably look into building cryptography 38 soon, but I provide no guarantee. In the long term, it would be great to have it as an official supported package.

[newfix]

Having Cryptography version 38 work with Chaquopy would be great.

Just out of curiosity: Is there any particular reason for choosing version 38 rather than the latest version?

[emanuele-f]

I'm trying to run the latest mitmproxy release, which strictly dependends on version 38 https://github.com/mitmproxy/mitmproxy/blob/9.0.1/setup.py#L79

[emanuele-f]

Hi @mhsmith, following the new instructions at https://github.com/chaquo/chaquopy/tree/master/server/pypi to build for python 3.10, I got the same error as before (e.g. see below for PyYAML). I think the problem is that python 3.8 must be used for the conda env (conda create -n build-wheel python=3.8) not python 3.10.

Building wheels for collected packages: PyYAML
  Building wheel for PyYAML (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [565 lines of output]
      In file included from ext/_yaml.c:271:
      ext/_yaml.h:10: warning: "PyString_CheckExact" redefined
         10 | #define PyString_CheckExact PyBytes_CheckExact
            |
      ext/_yaml.c:139: note: this is the location of the previous definition
        139 |   #define PyString_CheckExact          PyUnicode_CheckExact

After switching to python 3.8 in conda, pip install -r server/pypi/requirements.txt still fails. The problem seem to be in the PyYAML version and pip version. I've changed the requirements as follows:

-pip==19.2.3
-setuptools==46.4.0
-wheel==0.33.6
+#pip==19.2.3
+#setuptools==46.4.0
+#wheel==0.33.6

 # Other direct requirements
 Jinja2==2.10
 jsonschema==2.6.0
 pyelftools==0.24
-PyYAML==3.12
+PyYAML==3.13

and now installing them works good. I've then tried to build the brotli package just to see if things are set up properly, but it fails:

./build-wheel.py --python 3.10 --abi x86_64 brotli

build-wheel.py: mkdir -p /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel
build-wheel.py: unzip -d /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel -q /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/src/Brotli-1.0.7-cp310-cp310-linux_x86_64.whl
build-wheel.py: Processing native binaries
build-wheel.py: mv /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.cpython-310-x86_64-linux-gnu.so /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: chmod +w /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: /home/emanuele/Android/Sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip --strip-unneeded /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: Error: /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so is linked against unknown library 'libm.so.6'

Kindly request your help to proceed

[mhsmith]

Please post the full logs for both of those failures: you can attach them as files if they're too large to paste.

And please clarify the following:

• The first failure, with PyYAML: did that happen after the following sequence of commands? Or something different?

  conda create -n build-wheel python=3.10
  conda activate build-wheel
  pip install -r server/pypi/requirements.txt

• The second failure, with brotli: did you create that environment with Python 3.8 or 3.10?
  • If 3.8, it's apparently still managed to find a python3.10 executable somewhere to do the build, so where did that come from (which python3.10)?
  • And does the brotli build work with Python 3.8?

[emanuele-f]

I will redo on a clean environment and post the full logs. Do you confirm that to build for python 3.10 I have to use conda create -n build-wheel python=3.10 as of the docs?

[mhsmith]

Any python3.10 executable on the PATH should be fine: conda is just a convenient way of getting it.

[emanuele-f]

Mmm, but some tools that are used to build the wheels does not seem to support python 3.10 (for example, https://github.com/yaml/pyyaml/issues/416), so I cannot see how this can work, this is why I thought that some step was wrong, so tried using python 3.8 to run build-wheel.

Anyway, here are the steps following the server/pypi instructions, which lead to the error above (using latest chaquopy master):

cd chaquopy
mkdir build-clean
cd build-clean
mkdir -p sysroot overlay workdir
sudo mount -t overlay overlay -o lowerdir=/,upperdir=./sysroot,workdir=./workdir ./overlay
sudo arch-chroot ./overlay

cd /home/emanuele/src/chaquopy
mkdir -p maven/com/chaquo/python/target/3.10.6-1-1
cd maven/com/chaquo/python/target/3.10.6-1-1
wget https://repo.maven.apache.org/maven2/com/chaquo/python/target/3.10.6-1/target-3.10.6-1-x86_64.zip
conda create -n build-clean python=3.10
conda activate build-clean

# gives error, see attached `pip_install_error.txt`
pip install -r server/pypi/requirements.txt

pip_install_error.txt

The second failure, with brotli: did you create that environment with Python 3.8 or 3.10

It happened using python 3.8 in the conda env, but using --python 3.10 in build_sheel.py. I'm on archlinux and python3.10 is in my PATH. I thought build_wheel.py used the python version from the maven folder I've downloaded.

And does the brotli build work with Python 3.8

No. Following server/pypi instructions, it stills fails at the pip install step. Here is my commands:

mkdir -p maven/com/chaquo/python/target/3.8.16-0
cd maven/com/chaquo/python/target/3.8.16-0
wget https://repo.maven.apache.org/maven2/com/chaquo/python/target/3.8.16-0/target-3.8.16-0-x86_64.zip

cd /home/emanuele/src/chaquopy
conda create -n build-wheel python=3.8
conda activate build-wheel

# gives error, see attached `pip_38_install_error.txt`
pip install -r server/pypi/requirements.txt

In essence, without modifying the requirements.txt file, I cannot even reach the build step.

pip_38_install_error.txt

python --version: Python 3.8.16 which python: /opt/miniconda3/envs/build-wheel/bin/python pip --version pip 23.0.1 from /opt/miniconda3/envs/build-wheel/lib/python3.8/site-packages/pip (python 3.8)

[mhsmith]

I couldn't reproduce the PyYAML problem at first, because it looks like it falls back to a pure-Python mode when the build machine doesn't have yaml.h available. Anyway, I've now updated it to a version which has manylinux wheels on PyPI, so that shouldn't be a problem anymore. And I've updated several other requirements which also weren't compatible with Python 3.10. Thanks for reporting that.

So it should now be possible to run build-wheel with Python 3.8, 3.9 or 3.10. For Python packages it will then launch a subprocess to do the actual build, with a version matching your --python option.

[emanuele-f]

Hi Malcolm, thanks for the fixes. I confirm that building for python 3.10 works correctly now.

I can successfully build the new cryptography wheels, but I'm running into an issue with OpenSSL at runtime. The following error is thrown after cryptography is imported:

com.chaquo.python.PyException: InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=126615813, lib=15, reason=786693, reason_text=b'error:078C0105:common libcrypto routines::init fail')])

Here is the full log: openssl_error.txt

This is the wheel I've built that throws the above error: cryptography-38.0.4-0-cp310-cp310-android_21_x86.zip (rename to .whl). It uses cffi 1.15.1.

The error is thrown here: binding.py#L176. This is the relevant issue I've found: https://github.com/openssl/openssl/issues/19067 . It seems like OpenSSL is trying to load the "legacy" provider as a shared library via OSSL_PROVIDER_load, which fails (maybe because of the different libcrypto_chaquopy.so name?).

How can we address this?

[mhsmith]

Can you create a fork of this repository and push the files you used to build this wheel?

It would also be useful to include a README file which includes the instructions above, updated as necessary to reflect how you built this wheel.

[emanuele-f]

I've bypassed the above error by removing the _openssl_assert as explained in https://github.com/pyca/cryptography/issues/7358 . This is not optimal as it disables some ciphers, but for my use case it's fine.

I've uploaded the new wheels to https://github.com/emanuele-f/chaquopy-wheels . @newfix you can also find there the cryptography 39.0.2 modules cooked just for you ;) Please give it a try an let me know

I've quickly verified that the 38.0.4 wheels work correctly on x86 and arm64 in my PCAPdroid-mitm plugin. On armv7, however, I get ImportError: dlopen failed: cannot locate symbol "decode_eht_entry", seems related to libunwind (full log attached). @mhsmith tomorrow I will clean things up and post clean instructions, hopefully you will be able to reproduce the builds. If you have suggestions for the armv7 issue please let me know.

[mhsmith]

I've bypassed the above error by removing the _openssl_assert as explained in pyca/cryptography#7358 . This is not optimal as it disables some ciphers, but for my use case it's fine.

Thanks: there are some related notes about the OpenSSL legacy provider in #688.

[newfix]

I've uploaded the new wheels to https://github.com/emanuele-f/chaquopy-wheels . @newfix you can also find there the cryptography 39.0.2 modules cooked just for you ;) Please give it a try an let me know

Thank you very much for building this version!

I managed to install it. However, at runtime I get the following error: ImportError: dlopen failed: library "libpython3.10.so" not found: needed by .../hazmat/bindings/_rust.so in namespace classloader-namespace

Is this maybe because the briefcase toolchain I am using to build my application is referring to a different version (3.8) of python?

Any advice regarding how I could overcome the above error would be appreciated.

[emanuele-f]

Hi, yeah, all these wheel are only for python 3.10. You should consider upgrading your project to 3.10. I could try building for 3.8, but only after I fix the issues with armv7

[mhsmith]

@newfix: I'm not sure how you managed to install cp310 wheels into a Python 3.8 app. If you renamed them to cp38, that won't work, because different Python versions are not binary compatible.

When using Briefcase, the Python version used to run briefcase will determine the Python version of the app. So to use these wheels, you'll need to create a Python 3.10 environment, install Briefcase into it, then rerun briefcase create android.

[newfix]

Thank you for the feedback.

I did not rename the whl file. I just referenced it from my pyproject.toml file. After modifying my build.gradle file to restrict it to building the "arm64-v8a" abi I managed to build my app incl. the cryptography package. However, at runtime I am getting the error reported above.

I will set up a Python 3.10 environment as you suggested and try again.

[newfix]

Using Python 3.10 I got one step further, but now I am getting the following error:

RuntimeError: OpenSSL 3.0's legacy provider failed to load. This is a fatal error by default, but cryptography supports running without legacy algorithms by setting the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY. If you did not expect this error, you have likely made a mistake with your OpenSSL configuration.

I tried adding the following code snippet, but it did not resolve the issue:

import os
os.environ["CRYPTOGRAPHY_OPENSSL_NO_LEGACY"] = "1"

Any idea how I could solve this?

[newfix]

I just tried version 38 of the cryptography package and it worked :smiley:

So the issue with the legacy algorithms only exists for version 39.

[emanuele-f]

My mistake, should be fixed in the new cryptography 39 wheels I've just uploaded

[newfix]

Now, version 39 is also working fine :smiley:

@emanuele-f , @mhsmith : Thank you so much for your great work, for sharing it with others and for the support you are providing!

[emanuele-f]

Happy to share my work and save other people time :)

Pre-built wheels available at https://github.com/emanuele-f/chaquopy-wheels

Instructions and patches available at https://github.com/emanuele-f/chaquopy/blob/cryptography-wheel/README_cryptography_39.md

[emanuele-f]

@mhsmith regarding the issue with armv7, the problem is that the libpython that I build is linked against libunwind.a, whereas the chaquopy libpython is not.

$READELF -Wa armeabi-v7a/sysroot/usr/lib/libpython3.10.so | grep decode_eht_entry
002df850  0002cc16 R_ARM_JUMP_SLOT        002d4989   decode_eht_entry
   716: 002d4989    72 FUNC    GLOBAL DEFAULT    11 decode_eht_entry
 20136: 002d4989    72 FUNC    GLOBAL DEFAULT    11 decode_eht_entry

$READELF -Wa ./requirements/chaquopy/lib/libpython3.10.so | grep decode_eht_entry

$READELF -Wa ./fix_wheel/cryptography/hazmat/bindings/_rust.so | grep decode_eht_entry
0016e838  00008516 R_ARM_JUMP_SLOT        00000000   decode_eht_entry
   133: 00000000     0 FUNC    GLOBAL DEFAULT   UND decode_eht_entry

It seems like I'm forced to link against libunwind.a (Sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libunwind.a), otherwise armv7a-linux-androideabi21-clang fails to compile even a simple int main() {} source, failing with ld: error: unable to find library -lunwind. Do you know how to prevent armv7a-linux-androideabi21-clang to try to link with -lunwind? I'm using -Wl,--exclude-libs,libunwind.a in the LDFLAGS but it has no effect

Edit: small correction, as explained here, --exclude-libs does not prevent from linking but rather it gives symbols an hidden visibility

Edit2: I was able to workaround this issue by setting the -Wl,--exclude-libs,libunwind.a flag directly in the LDSHARED env and rebuilding python. Now the symbol decode_eht_entry is properly marked as local and the built wheel works correctly on armv7a!

$READELF -Wa libpython3.10.so | grep decode_eht_entry
 15390: 002d43a1    72 FUNC    LOCAL  DEFAULT    13 decode_eht_entry

I've upload the new armv7a wheels. Updating the cryptography build instructions.

Edit3: I've just realized that I mistyped LDFLAGS as LD_FLAGS 🤦‍♂️. This also fixes the issue

[mhsmith]

Thanks very much.

[Benoit-W]

Hello, I am currently trying to build a version of tokenizer (tokenizers!=0.11.3,<0.14,>=0.11.1) in order to use upper transformer version. As i saw on different issue it needs rust too. Do you think it is possible to do it with the information of this issue and the tutorial for building package (https://github.com/chaquo/chaquopy/tree/master/server/pypi) ?

[mhsmith]

Yes, it should be possible. In fact, the tokenizers recipe already has a patch for setting up the Rust toolchain, though I haven't tested it with the current version of the build tool.

See #1030 for current notes on Rust.