nabla-c0d3 / nassl

Experimental OpenSSL wrapper for Python 3.8+ and SSLyze.
GNU Affero General Public License v3.0
39 stars 35 forks source link

Alpine Linux: libssl.a: file format not recognized; treating as linker script #31

Closed petegallagher closed 6 years ago

petegallagher commented 6 years ago

Following on from #30 when trying to build nassl 1.1.0 on Alpine Linux I get the following error:

# sudo pip3 install --upgrade https://github.com/nabla-c0d3/nassl/archive/1.1.0.tar.gz
Collecting https://github.com/nabla-c0d3/nassl/archive/1.1.0.tar.gz
  Downloading https://github.com/nabla-c0d3/nassl/archive/1.1.0.tar.gz (2.0MB)
    100% |████████████████████████████████| 2.0MB 381kB/s
Installing collected packages: nassl
  Found existing installation: nassl 0.17.0
    Uninstalling nassl-0.17.0:
      Successfully uninstalled nassl-0.17.0
  Running setup.py install for nassl ... error
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-q2k5aflw-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-vs155ft9-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/nassl
    copying nassl/__init__.py -> build/lib.linux-x86_64-3.6/nassl
    copying nassl/ssl_client.py -> build/lib.linux-x86_64-3.6/nassl
    copying nassl/legacy_ssl_client.py -> build/lib.linux-x86_64-3.6/nassl
    copying nassl/ocsp_response.py -> build/lib.linux-x86_64-3.6/nassl
    running build_ext
    building 'nassl._nassl_legacy' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/nassl
    creating build/temp.linux-x86_64-3.6/nassl/_nassl
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_SSL_CTX.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_SSL_CTX.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_SSL.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_SSL.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_X509.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_X509.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_errors.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_errors.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_BIO.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_BIO.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_X509_EXTENSION.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_X509_EXTENSION.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_X509_NAME_ENTRY.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_X509_NAME_ENTRY.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_SSL_SESSION.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_SSL_SESSION.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/openssl_utils.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/openssl_utils.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/nassl_OCSP_RESPONSE.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_OCSP_RESPONSE.o -Wall
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DLEGACY_OPENSSL=1 -I/tmp/pip-q2k5aflw-build/bin/openssl-legacy/include -Inassl/_nassl -I/usr/include/python3.6m -c nassl/_nassl/python_utils.c -o build/temp.linux-x86_64-3.6/nassl/_nassl/python_utils.o -Wall
    gcc -shared -Wl,--as-needed -Wl,--as-needed build/temp.linux-x86_64-3.6/nassl/_nassl/nassl.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_SSL_CTX.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_SSL.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_X509.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_errors.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_BIO.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_X509_EXTENSION.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_X509_NAME_ENTRY.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_SSL_SESSION.o build/temp.linux-x86_64-3.6/nassl/_nassl/openssl_utils.o build/temp.linux-x86_64-3.6/nassl/_nassl/nassl_OCSP_RESPONSE.o build/temp.linux-x86_64-3.6/nassl/_nassl/python_utils.o /tmp/pip-q2k5aflw-build/bin/openssl-legacy/linux64/libssl.a /tmp/pip-q2k5aflw-build/bin/openssl-legacy/linux64/libcrypto.a /tmp/pip-q2k5aflw-build/bin/zlib/linux64/libz.a -L/usr/lib -lpython3.6m -o build/lib.linux-x86_64-3.6/nassl/_nassl_legacy.cpython-36m-x86_64-linux-gnu.so -Wl,-z,noexecstack
    /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld:/tmp/pip-q2k5aflw-build/bin/openssl-legacy/linux64/libssl.a: file format not recognized; treating as linker script
    /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld:/tmp/pip-q2k5aflw-build/bin/openssl-legacy/linux64/libssl.a:1: syntax error
    collect2: error: ld returned 1 exit status
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
  Rolling back uninstall of nassl
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-q2k5aflw-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-vs155ft9-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-q2k5aflw-build/

As a side-note pip is unable to get 1.1.0 from pypi because nassl-1.1.0.tar.gz is not available from https://pypi.python.org/simple/nassl/, so I had to provide the URL to github instead.

petegallagher commented 6 years ago

Just realised this is an issue with git-lfs rather than nassl specifically. So is it possible that you could provide a complete package for 1.1.0 on pypi containing the appropriate files (rather than references to lfs)?

nabla-c0d3 commented 6 years ago

Hello, I just pushed the 1.1.0 source package to pypi; should solve this problem.

waja commented 6 years ago

This seems to be still an issue.

Collecting nassl<1.2.0,>=1.1.0 (from sslyze==1.4.1)
  Could not find a version that satisfies the requirement nassl<1.2.0,>=1.1.0 (from sslyze==1.4.1) (from versions: 0.13.4.win32, 0.13.1, 0.13.2, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.14.0, 0.14.1, 0.14.2, 0.15.0, 0.15.1, 0.16.0, 0.16.1, 0.16.2, 0.16.3, 0.17.0, 1.0.1, 1.0.2, 1.0.3)
No matching distribution found for nassl<1.2.0,>=1.1.0 (from sslyze==1.4.1)
waja commented 6 years ago

On alpine:

/ # pip search nassl
nassl (1.1.0)  - Experimental OpenSSL wrapper for Python 2.7 / 3.4+ and SSLyze.
/ # pip install nassl==1.1.0
Collecting nassl==1.1.0
  Could not find a version that satisfies the requirement nassl==1.1.0 (from versions: 0.13.4.win32, 0.13.1, 0.13.2, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.14.0, 0.14.1, 0.14.2, 0.15.0, 0.15.1, 0.16.0, 0.16.1, 0.16.2, 0.16.3, 0.17.0, 1.0.1, 1.0.2, 1.0.3)
No matching distribution found for nassl==1.1.0
/ # pip download nassl
Collecting nassl
  Downloading nassl-1.0.3.tar.gz (30.0MB)
    100% |████████████████████████████████| 30.1MB 36kB/s 
  Saved /nassl-1.0.3.tar.gz
Collecting enum34 (from nassl)
  Downloading enum34-1.1.6-py2-none-any.whl
  Saved /enum34-1.1.6-py2-none-any.whl
Collecting typing (from nassl)
  Downloading typing-3.6.4-py2-none-any.whl
  Saved /typing-3.6.4-py2-none-any.whl
Successfully downloaded nassl enum34 typing
waja commented 6 years ago

Beside the download issue, the libssl.a:1: syntax error still exists (at least when downloading nassl 1.1.0 from github). See build log build.log for more details.

jsf9k commented 6 years ago

@waja I made some progress toward building the latest nassl from scratch on Alpine Linux. See here.

nabla-c0d3 commented 6 years ago

@waja The syntax error is due to Git LFS not being installed on your host.

waja commented 6 years ago

@nabla-c0d3 thanks for the info. But even with git lfs installed same problem. The syntax error is caused by a problem one line before:

/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld:/tmp/pip-RLBNvB-build/bin/openssl-legacy/linux64/libssl.a: file format not recognized; treating as linker script

This can be found in the build.log beside some more interesting part earlier.

waja commented 6 years ago

Could it be that https://github.com/pypa/pip/issues/3969 is the source of the issue that nassl isn't downloaded on alpine?

waja commented 6 years ago

Could it be that pypa/pip#3969 is the source of the issue that nassl isn't downloaded on alpine?

After working around the issue with the fix used in https://github.com/xemuliam/docker-python/blob/9a0154a07a8c2499eff0ed1f83f9fe857369b82b/Dockerfile#L24 the installation of the binaries works, but there is a (new) runtime issue:

Traceback (most recent call last):
  File "/usr/bin/sslyze", line 11, in <module>
    load_entry_point('SSLyze==1.4.1', 'console_scripts', 'sslyze')()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.6/site-packages/sslyze/__main__.py", line 10, in <module>
    from sslyze.plugins.plugin_base import PluginScanResult
  File "/usr/lib/python3.6/site-packages/sslyze/plugins/plugin_base.py", line 12, in <module>
    from sslyze.server_connectivity_info import ServerConnectivityInfo
  File "/usr/lib/python3.6/site-packages/sslyze/server_connectivity_info.py", line 6, in <module>
    from nassl.ssl_client import OpenSslVersionEnum
  File "/usr/lib/python3.6/site-packages/nassl/ssl_client.py", line 7, in <module>
    from nassl import _nassl  # type: ignore
ImportError: Error relocating /usr/lib/python3.6/site-packages/nassl/_nassl.cpython-36m-x86_64-linux-gnu.so: __register_atfork: symbol not found

I guess this is related to the fact, that the nassl package is prebuild on a glibc system. Anyway ... it is still failing with the official Docker python image with the alpine tag (python:alpine and even python:alpine3.7). My build process there is just:

apk --no-cache update && apk --no-cache upgrade && \
 apk --no-cache add openssl libstdc++ && \
 echo "manylinux1_compatible = True" > /usr/local/lib/python3.6/_manylinux.py && \
 pip install --upgrade sslyze==$SSLYZE_CLI_VERSION
nabla-c0d3 commented 6 years ago

Based on your link, the manylinux wheels will not work with alpine linux by default. I think the way to go is to build everything from scratch (#35).