search

timonwong / cyksuid

Fast Python implementation of KSUID (K-Sortable Globally Unique IDs) using Cython
BSD 3-Clause "New" or "Revised" License
52 stars 2 forks source link

Failed v2.0.0 compilation on Linux arm64 #26

Closed jalaziz closed 2 years ago

jalaziz commented 2 years ago

When installing cyksuid in an arm64 (aarch64) Linux docker image, the build fails:

DEBUG:docker_builder:#21 46.63   Building wheel for cyksuid (pyproject.toml): started
DEBUG:docker_builder:#21 46.80   Building wheel for cyksuid (pyproject.toml): finished with status 'error'
DEBUG:docker_builder:#21 46.81   error: subprocess-exited-with-error
DEBUG:docker_builder:#21 46.81
DEBUG:docker_builder:#21 46.81   × Building wheel for cyksuid (pyproject.toml) did not run successfully.
DEBUG:docker_builder:#21 46.81   │ exit code: 1
DEBUG:docker_builder:#21 46.81   ╰─> [97 lines of output]
DEBUG:docker_builder:#21 46.81       running bdist_wheel
DEBUG:docker_builder:#21 46.81       running build
DEBUG:docker_builder:#21 46.81       running build_py
DEBUG:docker_builder:#21 46.81       creating build
DEBUG:docker_builder:#21 46.81       creating build/lib.linux-aarch64-cpython-39
DEBUG:docker_builder:#21 46.81       creating build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/ksuid.py -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/__init__.py -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/__version__.py -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/hints.py -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/v2.py -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/fast_base62.pyx -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/_ksuid.pyx -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/fast_base62.pxd -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/_ksuid.pxd -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/_ksuid.pyi -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/fast_base62.pyi -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/py.typed -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/ksuidlite.h -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/cbase62.h -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/cbase62.cc -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/fast_base62.cpp -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       copying cyksuid/_ksuid.cpp -> build/lib.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       running build_ext
DEBUG:docker_builder:#21 46.81       building 'cyksuid.fast_base62' extension
DEBUG:docker_builder:#21 46.81       creating build/temp.linux-aarch64-cpython-39
DEBUG:docker_builder:#21 46.81       creating build/temp.linux-aarch64-cpython-39/cyksuid
DEBUG:docker_builder:#21 46.81       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c cyksuid/cbase62.cc -o build/temp.linux-aarch64-cpython-39/cyksuid/cbase62.o -Wno-write-strings -Wno-invalid-offsetof -Wno-sign-compare -Wno-unused-variable -std=c++14
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81          27 | };
DEBUG:docker_builder:#21 46.81             | ^
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       cyksuid/cbase62.cc:27:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘char’ [-Wnarrowing]
DEBUG:docker_builder:#21 46.81       error: command '/usr/bin/gcc' failed with exit code 1
DEBUG:docker_builder:#21 46.81       [end of output]
DEBUG:docker_builder:#21 46.81
DEBUG:docker_builder:#21 46.81   note: This error originates from a subprocess, and is likely not a problem with pip.
DEBUG:docker_builder:#21 46.81   ERROR: Failed building wheel for cyksuid

As a side note, it appears that the build is not generating wheels for Linux arm64.

jalaziz commented 2 years ago

So the issue has to do with arm64 defaulting to unsigned char. I pushed a fix here: https://github.com/jalaziz/cyksuid/commit/336ffc69a525a740dfb3cd9758ab988dd6b0fe55.

However, it then fails to build with gcc: error: cyksuid/fast_base62.cpp: No such file or directory (as expected due to the cython build only working via setup.py).

I then looked into fixing the github actions pipeline, but I see they don't offer arm64 runners. Seems like the workaround would be to handle the builds in docker images or ensure cross-compilation works on the ubuntu images.

I can try to look into that later.

jalaziz commented 2 years ago

I got aarch64 builds working on linux: https://github.com/jalaziz/cyksuid/tree/fix-arm64-build

That being said, it looks we end up building the CPP files twice, once by the setup.py call and then by cibuildwheel. However, the setup.py call is needed to cythonize the files.

timonwong commented 2 years ago

I then looked into fixing the github actions pipeline, but I see they don't offer arm64 runners. Seems like the workaround would be to handle the builds in docker images or ensure cross-compilation works on the ubuntu images.

The only way to build arm64 wheels is to use qemu, which is rather slow, it takes ~30min+ to finish:(

timonwong commented 2 years ago

Tests is addressed here: #28

timonwong commented 2 years ago

Or maybe I should use travis to build linux aarch64 wheels

timonwong commented 2 years ago

aarch64 wheels will be handled in #29

jalaziz commented 2 years ago

Thank you!