search

indygreg / PyOxidizer

A modern Python application packaging and distribution tool
Mozilla Public License 2.0
5.4k stars 234 forks source link

Unable to compile C module for ‘cryptography’ on MacOS 10.14.5 #68

Open bsiegel opened 5 years ago

bsiegel commented 5 years ago

It looks like it’s failing to find the OpenSSL headers. Interestingly I can run pip3 install "cryptography==2.7" --no-binary :all: and it does appear to compile fine locally when I do that. When PyOxidizer attempts to compile it however, I get the following error:

 Running setup.py install for cryptography: started
    Running setup.py install for cryptography: finished with status 'error'
    ERROR: Complete output from command /Users/bsiegel/Desktop/az/build/target/x86_64-apple-darwin/debug/pyoxidizer/python.6668202a3225/python/install/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/g9/828fv6h94kdf0_c9s91d75g80000gn/T/pip-install-5kjndael/cryptography/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/g9/828fv6h94kdf0_c9s91d75g80000gn/T/pip-record-oh9hegm7/install-record.txt --single-version-externally-managed --compile --home=/private/var/folders/g9/828fv6h94kdf0_c9s91d75g80000gn/T/pip-target-t5ls0hb5:
    ERROR: Could not find platform dependent libraries <exec_prefix>
    Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.9-x86_64-3.7
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography
    copying src/cryptography/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography
    copying src/cryptography/utils.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography
    copying src/cryptography/__about__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography
    copying src/cryptography/exceptions.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography
    copying src/cryptography/fernet.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat
    copying src/cryptography/hazmat/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat
    copying src/cryptography/hazmat/_oid.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    copying src/cryptography/x509/oid.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    copying src/cryptography/x509/ocsp.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    copying src/cryptography/x509/general_name.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    copying src/cryptography/x509/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    copying src/cryptography/x509/extensions.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    copying src/cryptography/x509/name.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    copying src/cryptography/x509/base.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    copying src/cryptography/x509/certificate_transparency.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/x509
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends
    copying src/cryptography/hazmat/backends/interfaces.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends
    copying src/cryptography/hazmat/backends/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    copying src/cryptography/hazmat/primitives/cmac.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    copying src/cryptography/hazmat/primitives/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    copying src/cryptography/hazmat/primitives/poly1305.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    copying src/cryptography/hazmat/primitives/constant_time.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    copying src/cryptography/hazmat/primitives/keywrap.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    copying src/cryptography/hazmat/primitives/hmac.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    copying src/cryptography/hazmat/primitives/hashes.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    copying src/cryptography/hazmat/primitives/padding.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/bindings
    copying src/cryptography/hazmat/bindings/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/bindings
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/x448.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/backend.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/ec.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/ciphers.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/ocsp.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/x509.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/aead.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/encode_asn1.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/rsa.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/dh.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/cmac.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/utils.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/poly1305.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/ed25519.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/dsa.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/decode_asn1.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/hmac.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/ed448.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/x25519.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    copying src/cryptography/hazmat/backends/openssl/hashes.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/backends/openssl
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/kdf
    copying src/cryptography/hazmat/primitives/kdf/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/kdf
    copying src/cryptography/hazmat/primitives/kdf/scrypt.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/kdf
    copying src/cryptography/hazmat/primitives/kdf/pbkdf2.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/kdf
    copying src/cryptography/hazmat/primitives/kdf/hkdf.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/kdf
    copying src/cryptography/hazmat/primitives/kdf/x963kdf.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/kdf
    copying src/cryptography/hazmat/primitives/kdf/kbkdf.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/kdf
    copying src/cryptography/hazmat/primitives/kdf/concatkdf.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/kdf
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/twofactor
    copying src/cryptography/hazmat/primitives/twofactor/totp.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/twofactor
    copying src/cryptography/hazmat/primitives/twofactor/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/twofactor
    copying src/cryptography/hazmat/primitives/twofactor/hotp.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/twofactor
    copying src/cryptography/hazmat/primitives/twofactor/utils.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/twofactor
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/serialization
    copying src/cryptography/hazmat/primitives/serialization/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/serialization
    copying src/cryptography/hazmat/primitives/serialization/pkcs12.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/serialization
    copying src/cryptography/hazmat/primitives/serialization/ssh.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/serialization
    copying src/cryptography/hazmat/primitives/serialization/base.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/serialization
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/ciphers
    copying src/cryptography/hazmat/primitives/ciphers/algorithms.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/ciphers
    copying src/cryptography/hazmat/primitives/ciphers/aead.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/ciphers
    copying src/cryptography/hazmat/primitives/ciphers/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/ciphers
    copying src/cryptography/hazmat/primitives/ciphers/modes.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/ciphers
    copying src/cryptography/hazmat/primitives/ciphers/base.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/ciphers
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/x448.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/ec.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/rsa.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/dh.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/utils.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/ed25519.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/dsa.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/ed448.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/x25519.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    copying src/cryptography/hazmat/primitives/asymmetric/padding.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/primitives/asymmetric
    creating build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/bindings/openssl
    copying src/cryptography/hazmat/bindings/openssl/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/bindings/openssl
    copying src/cryptography/hazmat/bindings/openssl/_conditional.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/bindings/openssl
    copying src/cryptography/hazmat/bindings/openssl/binding.py -> build/lib.macosx-10.9-x86_64-3.7/cryptography/hazmat/bindings/openssl
    running egg_info
    writing src/cryptography.egg-info/PKG-INFO
    writing dependency_links to src/cryptography.egg-info/dependency_links.txt
    writing requirements to src/cryptography.egg-info/requires.txt
    writing top-level names to src/cryptography.egg-info/top_level.txt
    reading manifest file 'src/cryptography.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    no previously-included directories found matching 'docs/_build'
    warning: no previously-included files found matching 'vectors'
    warning: no previously-included files matching '*' found under directory 'vectors'
    warning: no previously-included files found matching 'azure-pipelines.yml'
    warning: no previously-included files found matching '.azure-pipelines'
    warning: no previously-included files found matching '.travis.yml'
    warning: no previously-included files found matching '.travis'
    warning: no previously-included files matching '*' found under directory '.azure-pipelines'
    warning: no previously-included files matching '*' found under directory '.travis'
    warning: no previously-included files found matching 'release.py'
    warning: no previously-included files found matching '.coveragerc'
    warning: no previously-included files found matching 'codecov.yml'
    warning: no previously-included files found matching 'dev-requirements.txt'
    warning: no previously-included files found matching 'rtd-requirements.txt'
    warning: no previously-included files found matching 'tox.ini'
    writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
    running build_ext
    generating cffi module 'build/temp.macosx-10.9-x86_64-3.7/_padding.c'
    creating build/temp.macosx-10.9-x86_64-3.7
    generating cffi module 'build/temp.macosx-10.9-x86_64-3.7/_constant_time.c'
    generating cffi module 'build/temp.macosx-10.9-x86_64-3.7/_openssl.c'
    building '_openssl' extension
    creating build/temp.macosx-10.9-x86_64-3.7/build
    creating build/temp.macosx-10.9-x86_64-3.7/build/temp.macosx-10.9-x86_64-3.7
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/private/var/folders/dd/xb3jz0tj133_hgnvdttctwxc0000gn/T/tmphnt3aknw/deps/include -I/private/var/folders/dd/xb3jz0tj133_hgnvdttctwxc0000gn/T/tmphnt3aknw/deps/lib/libffi-3.2.1/include -I//private/var/folders/dd/xb3jz0tj133_hgnvdttctwxc0000gn/T/tmphnt3aknw/deps/include/ncurses -I/private/var/folders/dd/xb3jz0tj133_hgnvdttctwxc0000gn/T/tmphnt3aknw/deps/include/uuid -I/Users/bsiegel/Desktop/az/build/target/x86_64-apple-darwin/debug/pyoxidizer/python.6668202a3225/python/install/include/python3.7m -I/install/include/python3.7m -c build/temp.macosx-10.9-x86_64-3.7/_openssl.c -o build/temp.macosx-10.9-x86_64-3.7/build/temp.macosx-10.9-x86_64-3.7/_openssl.o -Wconversion -Wno-error=sign-conversion
    build/temp.macosx-10.9-x86_64-3.7/_openssl.c:498:10: fatal error: 'openssl/opensslv.h' file not found
    #include <openssl/opensslv.h>
             ^~~~~~~~~~~~~~~~~~~~
    1 error generated.
    error: command 'clang' failed with exit status 1
    ----------------------------------------
ERROR: Command "/Users/bsiegel/Desktop/az/build/target/x86_64-apple-darwin/debug/pyoxidizer/python.6668202a3225/python/install/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/g9/828fv6h94kdf0_c9s91d75g80000gn/T/pip-install-5kjndael/cryptography/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/g9/828fv6h94kdf0_c9s91d75g80000gn/T/pip-record-oh9hegm7/install-record.txt --single-version-externally-managed --compile --home=/private/var/folders/g9/828fv6h94kdf0_c9s91d75g80000gn/T/pip-target-t5ls0hb5" failed with error code 1 in /private/var/folders/g9/828fv6h94kdf0_c9s91d75g80000gn/T/pip-install-5kjndael/cryptography/
indygreg commented 5 years ago

I was going to say that your system is missing the OpenSSL headers. But getting a --no-binary :all: to work kinda disproves that theory.

I wonder if the OpenSSL headers location is getting picked up via a call to the Python distribution's sysconfig.get_config_var('OPENSSL_INCLUDES'). The Python distributions that PyOxidizer uses is advertising the OpenSSL path used in its build environment. We may want to unset this variable in the Python distribution so it doesn't get wrongfully used. Looking at the build log, I see -I paths that look like the paths that python-build-standalone uses...

(We should also consider shipping a build environment so all this stuff just works.)

A potential workaround might be to set OPENSSL_DIR or some other environment variable to force cryptography's build system to pick up a specific OpenSSL version. But this is definitely a bug in python-build-standalone that should be addressed.

indygreg commented 5 years ago

See also #26.

cvanlabe commented 5 years ago

I got it to build on my OS 10.14.5 machine with:

CFLAGS="-I$(brew --prefix openssl)/include" LDFLAGS="-L$(brew --prefix openssl)/lib" pyoxidizer build

But then when running it, got this:

Traceback (most recent call last):
  File "app", line 4, in <module>
  File "cryptography.fernet", line 18, in <module>
  File "cryptography.hazmat.primitives.padding", line 13, in <module>
ModuleNotFoundError: No module named 'cryptography.hazmat.bindings._padding'
SystemError

Which is probably the same issue as in #26 ?