search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.82k stars 2.13k forks source link

Docker image does not build for aarch64 since 1.18 #8077

Closed Legogris closed 4 years ago

Legogris commented 4 years ago

I'm pretty confident this was working for 1.17.

Getting the following when building docker image for arm64.

 > [linux/arm64 builder 7/7] RUN pip install --prefix="/install" --no-warn-script-location         /synapse[all]:
#13 4.536 Processing /synapse
#13 13.79 Collecting jsonschema>=2.5.1
#13 13.82   Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
#13 14.30 Processing /root/.cache/pip/wheels/68/17/69/ac196dd181e620bba5fae5488e4fd6366a7316dce13cf88776/frozendict-1.2-py3-none-any.whl
#13 14.44 Collecting unpaddedbase64>=1.1.0
#13 14.46   Using cached unpaddedbase64-1.1.0-py2.py3-none-any.whl (3.3 kB)
#13 14.82 Collecting canonicaljson>=1.1.3
#13 14.99   Downloading canonicaljson-1.2.0-py3-none-any.whl (8.5 kB)
#13 15.35 Collecting signedjson>=1.1.0
#13 15.38   Downloading signedjson-1.1.1.tar.gz (10 kB)                                                                                                                                                              #13 15.54   Installing build dependencies: started
#13 33.71   Installing build dependencies: finished with status 'done'
#13 33.71   Getting requirements to build wheel: started
#13 36.27   Getting requirements to build wheel: finished with status 'done'
#13 36.29   Installing backend dependencies: started
#13 57.79   Installing backend dependencies: finished with status 'done'
#13 57.80     Preparing wheel metadata: started
#13 61.68     Preparing wheel metadata: finished with status 'done'                                                                                                                                                  #13 62.53 Collecting pynacl>=1.2.1                                                                                                                                                                                   #13 62.57   Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)                                                                                                                                                                 #13 68.22   Installing build dependencies: started
#13 116.9   Installing build dependencies: finished with status 'error'
#13 116.9   ERROR: Command errored out with exit status 1:                                                                                                                                                           #13 116.9    command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wjxjqqh5/overlay --no-warn-script-location --no-binary :none:
 --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"''
#13 116.9        cwd: None
#13 116.9   Complete output (101 lines):
#13 116.9   Collecting setuptools>=40.8.0
#13 116.9     Using cached setuptools-49.3.2-py3-none-any.whl (790 kB)
#13 116.9   Collecting wheel
#13 116.9     Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
#13 116.9   Collecting cffi>=1.4.1
#13 116.9     Downloading cffi-1.14.1.tar.gz (468 kB)
#13 116.9   Collecting pycparser
#13 116.9     Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
#13 116.9   Building wheels for collected packages: cffi
#13 116.9     Building wheel for cffi (setup.py): started
#13 116.9     Building wheel for cffi (setup.py): finished with status 'error'
#13 116.9     ERROR: Command errored out with exit status 1:
#13 116.9      command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-9hs8ewcc/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-9hs8ewcc/cffi/setup.py'"'"';f=
getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-p91sgoou
#13 116.9          cwd: /tmp/pip-install-9hs8ewcc/cffi/
#13 116.9     Complete output (36 lines):
#13 116.9     running bdist_wheel
#13 116.9     running build
#13 116.9     running build_py
#13 116.9     creating build
#13 116.9     creating build/lib.linux-aarch64-3.7
#13 116.9     creating build/lib.linux-aarch64-3.7/cffi
#13 116.9     copying cffi/commontypes.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9     copying cffi/model.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9     copying cffi/error.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9     copying cffi/__init__.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9     copying cffi/vengine_gen.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9     copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/cparser.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/pkgconfig.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/verifier.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/recompiler.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/lock.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/api.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/ffiplatform.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/_cffi_include.h -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/parse_c_type.h -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/_embedding.h -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-3.7/cffi
#13 116.9       running build_ext
#13 116.9       building '_cffi_backend' extension
#13 116.9       creating build/temp.linux-aarch64-3.7
#13 116.9       creating build/temp.linux-aarch64-3.7/c
#13 116.9       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-aarch64-3.7/c/_cffi_backend.o
#13 116.9       c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
#13 116.9        #include <ffi.h>
#13 116.9                 ^~~~~~~
#13 116.9       compilation terminated.
#13 116.9       error: command 'gcc' failed with exit status 1
#13 116.9       ----------------------------------------
#13 116.9   ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-9hs8ewcc/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-9hs8ewcc/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-gf_3eh38/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-wjxjqqh5/overlay --compile --install-headers /tmp/pip-build-env-wjxjqqh5/overlay/include/python3.7m/cffi Check the logs for full command output.
#13 116.9   ----------------------------------------
#13 116.9 ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wjxjqqh5/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
------
clokep commented 4 years ago

I don't believe a dependency on cffi has changed between 1.17 and 1.18. It seems that you're missing some headers though, the directions for installing cffi might help: https://cffi.readthedocs.io/en/latest/installation.html

(Looks like you need the python-dev and libffi-dev packages?)

Legogris commented 4 years ago

@clokep Note that this is just building the container image from /docker/Dockerfile, which I assume should be self-contained. I'd wager this may be related to https://github.com/matrix-org/synapse/pull/7839. Busy now but may look into making a PR at some point in the future unless anyone else addresses it.

clokep commented 4 years ago

I somehow totally missed the word "Docker" 😄

Yes, it is likely related to that.

richvdh commented 4 years ago

I think this has been addressed in https://github.com/matrix-org/synapse/pull/7921, although that PR seems to add more libraries than are necessary. A PR which fixes specifically this problem (independently of multiarch support) would be appreciated.

Legogris commented 4 years ago

I think this has been addressed in #7921, although that PR seems to add more libraries than are necessary. A PR which fixes specifically this problem (independently of multiarch support) would be appreciated.

This is precisely multi-arch support I'm raising here, actually. Can't comment on if either of those specific libraries are required, though.

maquis196 commented 4 years ago

Yeah ill raise a pr to get this working on other archs whilst im getting the rest of the multi-arch docker stuff through.

maquis196 commented 4 years ago

So ive raise https://github.com/matrix-org/synapse/pull/8144, with all the packages I needed to build the docker image on my pi. I'm not sure why they arent needed on amd64, I assume the base image is different somehow.

My multi-arch branch needed even more dependencies, so could be aarch64 might still fail, i could put them all in to be on the safe side or @Legogris , could you try the dockerfile and confirm if it works or not before we merge?

maquis196 commented 4 years ago

actually, since the delta between this and the multi-arch pr was just libxml, ive added it in to be on the safe side, its only the build image and a binary so cost is negligible in terms of time.