search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.82k stars 2.13k forks source link

v1.10.0rc5: Docker build fails with "gcc: fatal error: Killed signal terminated program cc1" #6889

Closed schildbach closed 4 years ago

schildbach commented 4 years ago

On an Ubuntu 18.04.4 LTS with the standard docker.io package installed, I ran

docker build -t my/synapse -f docker/Dockerfile .

and I got this:

Sending build context to Docker daemon  5.462MB
Step 1/16 : ARG PYTHON_VERSION=3.7
Step 2/16 : FROM docker.io/python:${PYTHON_VERSION}-alpine3.10 as builder
 ---> 64442cbbbbc3
Step 3/16 : RUN apk add         build-base         libffi-dev         libjpeg-turbo-dev         libressl-dev         libxslt-dev         linux-headers         postgresql-dev         zlib-dev
 ---> Using cache
 ---> 9c6e854b2272
Step 4/16 : RUN pip install --prefix="/install" --no-warn-script-location         cryptography         msgpack-python         pillow         pynacl
 ---> Using cache
 ---> 301cd2f1800a
Step 5/16 : COPY synapse /synapse/synapse/
 ---> 969d325786f1
Step 6/16 : COPY scripts /synapse/scripts/
 ---> cccab737a757
Step 7/16 : COPY MANIFEST.in README.rst setup.py synctl /synapse/
 ---> b12311f1d241
Step 8/16 : RUN pip install --prefix="/install" --no-warn-script-location         /synapse[all]
 ---> Running in 3be04b98d6b6
Processing /synapse
Collecting jsonschema>=2.5.1
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting frozendict>=1
  Downloading frozendict-1.2.tar.gz (2.6 kB)
Collecting unpaddedbase64>=1.1.0
  Downloading unpaddedbase64-1.1.0-py2.py3-none-any.whl (3.3 kB)
Collecting canonicaljson>=1.1.3
  Downloading canonicaljson-1.1.4-py2.py3-none-any.whl (5.9 kB)
Collecting signedjson>=1.1.0
  Downloading signedjson-1.1.tar.gz (10 kB)
Processing /root/.cache/pip/wheels/96/4b/45/e2ed4df10e87eaee077c4a0e0274107a683da06713bd0bcbfc/PyNaCl-1.3.0-cp37-cp37m-linux_x86_64.whl
Collecting idna>=2.5
  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)
Collecting service_identity>=18.1.0
  Downloading service_identity-18.1.0-py2.py3-none-any.whl (11 kB)
Collecting Twisted>=18.9.0
  Downloading Twisted-19.10.0.tar.bz2 (3.1 MB)
Collecting treq>=15.1
  Downloading treq-18.6.0-py2.py3-none-any.whl (53 kB)
Collecting pyopenssl>=16.0.0
  Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
Collecting pyyaml>=3.11
  Downloading PyYAML-5.3.tar.gz (268 kB)
Collecting pyasn1>=0.1.9
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.7
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting daemonize>=2.3.1
  Downloading daemonize-2.5.0-py2.py3-none-any.whl (5.2 kB)
Collecting bcrypt>=3.1.0
  Downloading bcrypt-3.1.7.tar.gz (42 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Processing /root/.cache/pip/wheels/33/7a/37/aa7173ed9589c201e1e99a9338af5b1c6526968a377dc473f1/Pillow-7.0.0-cp37-cp37m-linux_x86_64.whl
Collecting sortedcontainers>=1.4.4
  Downloading sortedcontainers-2.1.0-py2.py3-none-any.whl (28 kB)
Collecting pymacaroons>=0.13.0
  Downloading pymacaroons-0.13.0-py2.py3-none-any.whl (19 kB)
Collecting msgpack>=0.5.2
  Downloading msgpack-0.6.2.tar.gz (119 kB)
Collecting phonenumbers>=8.2.0
  Downloading phonenumbers-8.11.3-py2.py3-none-any.whl (2.6 MB)
Collecting six>=1.10
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting prometheus_client<0.8.0,>=0.0.18
  Downloading prometheus_client-0.7.1.tar.gz (38 kB)
Collecting attrs>=17.4.0
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting netaddr>=0.7.18
  Downloading netaddr-0.7.19-py2.py3-none-any.whl (1.6 MB)
Collecting Jinja2>=2.9
  Downloading Jinja2-2.11.1-py2.py3-none-any.whl (126 kB)
Collecting bleach>=1.4.3
  Downloading bleach-3.1.0-py2.py3-none-any.whl (157 kB)
Collecting typing-extensions>=3.7.4
  Downloading typing_extensions-3.7.4.1-py3-none-any.whl (20 kB)
Collecting jaeger-client>=4.0.0
  Downloading jaeger-client-4.3.0.tar.gz (81 kB)
Collecting pysaml2>=4.5.0
  Downloading pysaml2-5.0.0-py2.py3-none-any.whl (336 kB)
Collecting sentry-sdk>=0.7.2
  Downloading sentry_sdk-0.14.1-py2.py3-none-any.whl (93 kB)
Collecting lxml>=3.5.0
  Downloading lxml-4.5.0.tar.gz (4.5 MB)
Collecting txacme>=0.9.2
  Downloading txacme-0.9.2-py2.py3-none-any.whl (62 kB)
Collecting parameterized
  Downloading parameterized-0.7.1-py2.py3-none-any.whl (24 kB)
Collecting psycopg2>=2.7
  Downloading psycopg2-2.8.4.tar.gz (377 kB)
Collecting mock>=2.0
  Downloading mock-4.0.1-py3-none-any.whl (28 kB)
Collecting opentracing>=2.2.0
  Downloading opentracing-2.3.0.tar.gz (48 kB)
Collecting matrix-synapse-ldap3>=0.1
  Downloading matrix-synapse-ldap3-0.1.4.tar.gz (12 kB)
Collecting pyjwt>=1.6.4
  Downloading PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting pyrsistent>=0.14.0
  Downloading pyrsistent-0.15.7.tar.gz (107 kB)
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from jsonschema>=2.5.1->matrix-synapse==1.10.0rc5) (45.1.0)
Collecting simplejson>=3.6.5
  Downloading simplejson-3.17.0.tar.gz (83 kB)
Processing /root/.cache/pip/wheels/c4/91/4f/81fe591804590e3cd010d6c54f251607dc4c61fc5eb5840a51/cffi-1.14.0-cp37-cp37m-linux_x86_64.whl
  Downloading zope.interface-4.7.1.tar.gz (151 kB)
Collecting constantly>=15.1
  Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting incremental>=16.10.1
  Using cached incremental-17.5.0-py2.py3-none-any.whl (16 kB)
Collecting Automat>=0.3.0
  Downloading Automat-0.8.0-py2.py3-none-any.whl (31 kB)
Collecting hyperlink>=17.1.1
  Downloading hyperlink-19.0.0-py2.py3-none-any.whl (38 kB)
Collecting PyHamcrest>=1.9.0
  Downloading PyHamcrest-2.0.0-py3-none-any.whl (51 kB)
Collecting requests>=2.1.0
  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
Collecting MarkupSafe>=0.23
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting threadloop<2,>=1
  Downloading threadloop-1.0.2.tar.gz (4.9 kB)
Collecting thrift
  Downloading thrift-0.13.0.tar.gz (59 kB)
Collecting tornado>=4.3
  Downloading tornado-6.0.3.tar.gz (482 kB)
Collecting python-dateutil
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting defusedxml
  Downloading defusedxml-0.6.0-py2.py3-none-any.whl (23 kB)
Collecting pytz
  Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting certifi
  Downloading certifi-2019.11.28-py2.py3-none-any.whl (156 kB)
Collecting urllib3>=1.10.0
  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Collecting eliot>=0.8.0
  Downloading eliot-1.12.0-py2.py3-none-any.whl (113 kB)
Collecting pem>=16.1.0
  Downloading pem-20.1.0-py2.py3-none-any.whl (8.5 kB)
  copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/defs.py -> build/lib.linux-x86_64-3.7/lxml/html
  copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-3.7/lxml/html
  creating build/lib.linux-x86_64-3.7/lxml/isoschematron
  copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-3.7/lxml/isoschematron
  copying src/lxml/etree.h -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/etree_api.h -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-3.7/lxml
  copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/__init__.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-3.7/lxml/includes
  copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-3.7/lxml/includes
  creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources
  creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl
  copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
  running build_ext
ludes -I/usr/local/include/python3.7m -c src/lxml/etree.c -o build/temp.linux-x86_64-3.7/src/lxml/etree.o -w
    copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-3.7/lxml/html
    creating build/lib.linux-x86_64-3.7/lxml/isoschematron
    copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-3.7/lxml/isoschematron
    copying src/lxml/etree.h -> build/lib.linux-x86_64-3.7/lxml
    copying src/lxml/etree_api.h -> build/lib.linux-x86_64-3.7/lxml
    copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-3.7/lxml
    copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-3.7/lxml
    copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/__init__.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-3.7/lxml/includes
    copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-3.7/lxml/includes
    creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources
    creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/rng
    copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/rng
    creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl
    creating build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-3.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    running build_ext
    building 'lxml.etree' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/src
    creating build/temp.linux-x86_64-3.7/src/lxml
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -I/usr/include/libxml2 -Isrc -Isrc/lxml/includes -I/usr/local/include/python3.7m -c src/lxml/etree.c -o build/temp.linux-x86_64-3.7/src/lxml/etree.o -w
    gcc: fatal error: Killed signal terminated program cc1
    compilation terminated.
    Compile failed: command 'gcc' failed with exit status 1
    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitdr6435a4.c -o tmp/xmlXPathInitdr6435a4.o
    cc tmp/xmlXPathInitdr6435a4.o -L/usr/lib -lxml2 -o a.out
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-owk2zrhq/lxml/setup.py'"'"'; __file__='"'"'/tmp/pip-install-owk2zrhq/lxml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-5idgu1w_/install-record.txt --single-version-externally-managed --prefix /install --compile --install-headers /install/include/python3.7m/lxml Check the logs for full command output.
The command '/bin/sh -c pip install --prefix="/install" --no-warn-script-location         /synapse[all]' returned a non-zero code: 1

Since libxml is mentioned in the last messages, I wonder if libxml has been updated to the fixed version? See CVE-2019-19956, CVE-2020-7595.

richvdh commented 4 years ago

I don't think either of those CVEs should cause your compiler to be killed. perhaps it's just running out of memory?

In any case, we don't ship libxml; that is the responsibility of the base OS: Alpine Linux in the case of this docker build.

schildbach commented 4 years ago

Well, don't you inherit this responsibility by depending on some base OS version? So my question becomes: has the base OS version been updated to a version that doesn't come with the mentioned security flaws?

richvdh commented 4 years ago

I thought your question was about why your docker build fails? If it's not, then... your report is very confusing.

Our stock docker images use libxml 2.9.9, which predates both the advisories you mention. Indeed, CVE-2020-7595 applies to libxml 2.9.10, which is the latest release of libxml2.

To my knowledge, neither of the problems are exploitable in synapse. If you know otherwise, I would urge you to follow our security disclosure policy by reporting it to security@matrix.org rather than discussing it further in an open forum.

schildbach commented 4 years ago

No, it's fine. I was just a bit puzzled by your "security is responsibility of others" type of response. I'll close this and check with the next RC if the failure might have been caused by OOM.

richvdh commented 4 years ago

(for the record: we're bumping alpine linux to 3.11 for the 1.10.0 release: #6897)