search

squeaky-pl / portable-pypy

Portable 64 bit x86 PyPy binaries for many Linux distributions.
Other
478 stars 38 forks source link

2.5+ missing third party library headers - cryptography compiling error #9

Closed posita closed 9 years ago

posita commented 9 years ago

I first noticed this while continuing to investigate pyca/pyopenssl#135 (originally motivated by #2; see also this).

Here's how to reproduce:

# rm -fr /opt/site
# mkdir /opt/site
# .../pypy-2.5.1-linux_x86_64-portable/bin/virtualenv-pypy /opt/site
...
# /opt/site/bin/pip install pyOpenSSL
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pyOpenSSL
  Downloading pyOpenSSL-0.14.tar.gz (128kB)
    100% |################################| 131kB 617kB/s
Collecting cryptography>=0.2.1 (from pyOpenSSL)
  Downloading cryptography-0.8.1.tar.gz (268kB)
    100% |################################| 270kB 579kB/s
Collecting six>=1.5.2 (from pyOpenSSL)
  Downloading six-1.9.0-py2.py3-none-any.whl
Collecting pyasn1 (from cryptography>=0.2.1->pyOpenSSL)
  Downloading pyasn1-0.1.7.tar.gz (68kB)
    100% |################################| 69kB 820kB/s
Requirement already satisfied (use --upgrade to upgrade): setuptools in /opt/site/site-packages (from cryptography>=0.2.1->pyOpenSSL)
Collecting enum34 (from cryptography>=0.2.1->pyOpenSSL)
  Downloading enum34-1.0.4.tar.gz
Installing collected packages: enum34, pyasn1, six, cryptography, pyOpenSSL
  Running setup.py install for enum34
  Running setup.py install for pyasn1

  Running setup.py install for cryptography
    building '_Cryptography_cffi_f3e4673fx399b1113' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o
    cc -shared build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o -lssl -lcrypto -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
    building '_Cryptography_cffi_590da19fxffc7b1ce' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o
    cc -shared build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
    building '_Cryptography_cffi_26cb75b8x62b488b1' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o
    cc -shared build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
  Running setup.py install for pyOpenSSL
Successfully installed cryptography-0.8.1 enum34-1.0.4 pyOpenSSL-0.14 pyasn1-0.1.7 six-1.9.0
# /opt/site/bin/pypy
Python 2.7.9 (af9d5af2e8e35e39a6dd7266becb8eb08926cab4, Mar 30 2015, 17:13:10)
[PyPy 2.5.1 with GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> import OpenSSL.SSL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/site/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/opt/site/site-packages/OpenSSL/rand.py", line 11, in <module>
    from OpenSSL._util import (
  File "/opt/site/site-packages/OpenSSL/_util.py", line 4, in <module>
    binding = Binding()
  File "/opt/site/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 113, in __init__
    self._ensure_ffi_initialized()
  File "/opt/site/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 123, in _ensure_ffi_initialized
    cls._modules,
  File "/opt/site/site-packages/cryptography/hazmat/bindings/utils.py", line 31, in load_library_for_binding
    lib = ffi.verifier.load_library()
  File ".../files/pypy-2.5.1-linux_x86_64-portable/lib_pypy/cffi/verifier.py", line 97, in load_library
    return self._load_library()
  File ".../files/pypy-2.5.1-linux_x86_64-portable/lib_pypy/cffi/verifier.py", line 207, in _load_library
    return self._vengine.load_library()
  File ".../files/pypy-2.5.1-linux_x86_64-portable/lib_pypy/cffi/vengine_gen.py", line 66, in load_library
    module = backend.load_library(filename, flags)
OSError: Cannot load library /opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so: /opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so: symbol EC_GFp_nistp521_method, version OPENSSL_1.0.1 not defined in file libcrypto.so.1.0.0 with link time reference

Note this does not happen with 2.3.1:

# rm -fr /opt/site
# mkdir /opt/site
# .../pypy-2.3.1-linux_x86_64-portable/bin/virtualenv-pypy /opt/site
...
# /opt/site/bin/pip install pyOpenSSL
...
# /opt/site/bin/pypy
Python 2.7.6 (32f35069a16d819b58c1b6efb17c44e3e53397b2, Jun 10 2014, 00:42:27)
[PyPy 2.3.1 with GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> import OpenSSL.SSL
>>>> '{:x}'.format(OpenSSL.SSL.OPENSSL_VERSION_NUMBER)
'1000105f'
>>>> exit()
posita commented 9 years ago

I've now tested the following builds:

Build Result
pypy-2.5.1-linux_x86_64-portable :no_entry: _failure_
pypy-2.5-linux_x86_64-portable :no_entry: _failure_
pypy3-2.4-linux_x86_64-portable :white_check_mark: success
pypy-2.4-linux_x86_64-portable :white_check_mark: success
pypy3-2.3.1-linux_x86_64-portable :white_check_mark: success
pypy-2.3.1-linux_x86_64-portable :white_check_mark: success

For each, I did the following:

rm -fr /opt/site
mkdir /opt/site
.../[VERSION]-linux_x86_64-portable/bin/virtualenv-pypy /opt/site
/opt/site/bin/pip install pyOpenSSL
/opt/site/bin/pypy -c 'import OpenSSL.SSL ; print("{:x}".format(OpenSSL.SSL.OPENSSL_VERSION_NUMBER))'

Success indicates the OpenSSL version number was printed. Failure indicates a linking exception was raised.

squeaky-pl commented 9 years ago

One reason this might happen is that previously OpenSSL was statically linked into my pypy binary, since 2.5 upstream pypy by default split it's binary into small executable and large shared object, and this large shared object references other shared libraries (including libcrypto and libssl I ship) that are included in the portable PyPy binary and live inside lib folder. 

ldd pypy
       ....
        libpypy-c.so => /home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/bin/./libpypy-c.so (0x00007f943d75c000)
        ...

and that libpypy-c.so references libraries inside lib

ldd libpypy-c.so 
        ....
        libssl.so.1.0.0 => /home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/bin/./../lib/libssl.so.1.0.0 (0x00007fa75fac3000)
        libcrypto.so.1.0.0 => /home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/bin/./../lib/libcrypto.so.1.0.0 (0x00007fa75f671000)
        libexpat.so.1 => /home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/bin/./../lib/libexpat.so.1 (0x00007fa75f43e000)
        libffi.so.6 => /home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/bin/./../lib/libffi.so.6 (0x00007fa75edf2000)
       ...

https://github.com/squeaky-pl/portable-pypy/blob/docker/build_deps#L15 As you can see here this version of portable pypy is built against version 1.0.2 of openssl (and that is where libssl.1.0.0 and libcrypto.so.1.0.0 come, dont get confused by file names). Your linking error mentions version OPENSSL_1.0.1

Can you try running ldd /_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so to find out whichlibcrypto the dynamic loader will try to load? My guess is that it is linked against system-wide libcrypto but since in the process space of portable PyPy there is already another version of libssl and libcrypto (the one I ship) with possibly other symbols defined/undefined conflict happens.

posita commented 9 years ago

I should have thought of that. :blush: It looks like you're right:

# find /opt/site -name _Cryptography_cffi_\*.pypy-25.so
/opt/site/site-packages/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
/opt/site/site-packages/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
/opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
# for i in $( find /opt/site -name _Cryptography_cffi_\*.pypy-25.so ) ; do echo "${i}" ; ldd "${i}" ; done
/opt/site/site-packages/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
    linux-vdso.so.1 =>  (0x00007fffa2dd3000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbe4308a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fbe4361e000)
/opt/site/site-packages/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
    linux-vdso.so.1 =>  (0x00007fff5fd8e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f907a6f9000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f907ac8d000)
/opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
    linux-vdso.so.1 =>  (0x00007ffe211de000)
    libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fa89cd14000)
    libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fa89c91c000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa89c590000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa89c38c000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa89c175000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa89d1c3000)

I should have paid more attention to the build commands that resulted from /opt/site/bin/pip install pyOpenSSL (which I omitted from my original post, but have now gone back and added). I noticed that the compile commands properly add -I/opt/site/include, but there is no such corresponding -L... argument accompanying the respective linking phases.

posita commented 9 years ago

Hmmm...I might have spoke too soon regarding -L.... I tried this (without success):

# /opt/site/bin/pip uninstall cryptography
...
  Successfully uninstalled cryptography-0.8.1
# find /opt/site -name _Cryptography_cffi_\*.pypy-25.so
# LDFLAGS='-L/opt/site/lib' /opt/site/bin/pip install cryptography
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting cryptography
  Using cached cryptography-0.8.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /opt/site/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /opt/site/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /opt/site/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): enum34 in /opt/site/site-packages (from cryptography)
Installing collected packages: cryptography
  Running setup.py install for cryptography
    building '_Cryptography_cffi_f3e4673fx399b1113' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o
    cc -shared -L/opt/site/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o -lssl -lcrypto -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
    building '_Cryptography_cffi_590da19fxffc7b1ce' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o
    cc -shared -L/opt/site/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
    building '_Cryptography_cffi_26cb75b8x62b488b1' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o
    cc -shared -L/opt/site/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
Successfully installed cryptography-0.8.1
# for i in $( find /opt/site -name _Cryptography_cffi_\*.pypy-25.so ) ; do echo "${i}" ; ldd "${i}" ; done
/opt/site/site-packages/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
    linux-vdso.so.1 =>  (0x00007fff0b9e9000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007faf87547000)
    /lib64/ld-linux-x86-64.so.2 (0x00007faf87adb000)
/opt/site/site-packages/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
    linux-vdso.so.1 =>  (0x00007ffe76382000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fee341db000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fee3476f000)
/opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
    linux-vdso.so.1 =>  (0x00007ffd817e5000)
    libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f3a40955000)
    libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f3a4055d000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3a401d1000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3a3ffcd000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3a3fdb6000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3a40e04000)
posita commented 9 years ago

UPDATE: see following comment.

I spoke too soon about speaking too soon. It works after building with LDFLAGS='/opt/site/lib' (irrespective of what ldd says):

# LD_TRACE_LOADED_OBJECTS=all /opt/site/bin/pypy -c 'import OpenSSL.SSL ; print("{:x}".format(OpenSSL.SSL.OPENSSL_VERSION_NUMBER))'
    linux-vdso.so.1 =>  (0x00007ffe07dd7000)
    libpypy-c.so => /opt/site/bin/libpypy-c.so (0x00007fb5c6567000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb5c634b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb5c5fc0000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb5c5dbc000)
    libbz2.so.1 => /lib/x86_64-linux-gnu/libbz2.so.1 (0x00007fb5c5bac000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb5c592a000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb5c5713000)
    libssl.so.1.0.0 => /opt/site/bin/../lib/libssl.so.1.0.0 (0x00007fb5c549b000)
    libcrypto.so.1.0.0 => /opt/site/bin/../lib/libcrypto.so.1.0.0 (0x00007fb5c5049000)
    libexpat.so.1 => /opt/site/bin/../lib/libexpat.so.1 (0x00007fb5c4e16000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fb5c4bdf000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fb5c49d7000)
    libffi.so.6 => /opt/site/bin/../lib/libffi.so.6 (0x00007fb5c47ca000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fb5c45c7000)
    libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007fb5c43a5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb5cb2b2000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fb5c417c000)
# /opt/site/bin/pypy -c 'import OpenSSL.SSL ; print("{:x}".format(OpenSSL.SSL.OPENSSL_VERSION_NUMBER))'
1000105f

So is this a cffi problem? A pip problem? Both? Neither? :confused:

posita commented 9 years ago

Speaking of speaking too soon, now I can't reproduce my previous https://github.com/squeaky-pl/portable-pypy/issues/9#issuecomment-91080140. I must have done something weird that I wasn't aware of. :confounded:

cffi and cryptography are some stubborn customers (I tried wading through this, but I didn't find it much help):

# rm -fr /opt/site/* ~/.cache ~/.pip && .../pypy-2.5.1-linux_x86_64-portable/bin/virtualenv-pypy /opt/site
New pypy executable in /opt/site/bin/pypy
Installing setuptools, pip...done.
# env LIBRARY_PATH=/opt/site/lib LDFLAGS='-L/opt/site/lib -v' /opt/site/bin/pip install pyOpenSSL
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pyOpenSSL
  Downloading pyOpenSSL-0.14.tar.gz (128kB)
    100% |################################| 131kB 316kB/s
Collecting cryptography>=0.2.1 (from pyOpenSSL)
  Downloading cryptography-0.8.1.tar.gz (268kB)
    100% |################################| 270kB 486kB/s
Collecting six>=1.5.2 (from pyOpenSSL)
  Downloading six-1.9.0-py2.py3-none-any.whl
Collecting pyasn1 (from cryptography>=0.2.1->pyOpenSSL)
  Downloading pyasn1-0.1.7.tar.gz (68kB)
    100% |################################| 69kB 651kB/s
Requirement already satisfied (use --upgrade to upgrade): setuptools in /opt/site/site-packages (from cryptography>=0.2.1->pyOpenSSL)
Collecting enum34 (from cryptography>=0.2.1->pyOpenSSL)
  Downloading enum34-1.0.4.tar.gz
Installing collected packages: enum34, pyasn1, six, cryptography, pyOpenSSL
  Running setup.py install for enum34
  Running setup.py install for pyasn1

  Running setup.py install for cryptography
    building '_Cryptography_cffi_f3e4673fx399b1113' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o
    cc -shared -L/opt/site/lib -v build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o -lssl -lcrypto -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
    Using built-in specs.
    COLLECT_GCC=cc
    COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
    Target: x86_64-linux-gnu
    Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
    Thread model: posix
    gcc version 4.7.2 (Debian 4.7.2-5)
    COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/
    LIBRARY_PATH=/opt/site/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/opt/site/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../:/lib/:/usr/lib/
    COLLECT_GCC_OPTIONS='-shared' '-L/opt/site/lib' '-v' '-o' 'build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so' '-mtune=generic' '-march=x86-64'
     /usr/lib/gcc/x86_64-linux-gnu/4.7/collect2 --sysroot=/ --build-id --no-add-needed --eh-frame-hdr -m elf_x86_64 --hash-style=both -shared -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o -L/opt/site/lib -L/opt/site/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7 -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/opt/site/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o -lssl -lcrypto -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o
    building '_Cryptography_cffi_590da19fxffc7b1ce' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o
    cc -shared -L/opt/site/lib -v build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
    Using built-in specs.
    COLLECT_GCC=cc
    COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
    Target: x86_64-linux-gnu
    Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
    Thread model: posix
    gcc version 4.7.2 (Debian 4.7.2-5)
    COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/
    LIBRARY_PATH=/opt/site/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/opt/site/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../:/lib/:/usr/lib/
    COLLECT_GCC_OPTIONS='-shared' '-L/opt/site/lib' '-v' '-o' 'build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so' '-mtune=generic' '-march=x86-64'
     /usr/lib/gcc/x86_64-linux-gnu/4.7/collect2 --sysroot=/ --build-id --no-add-needed --eh-frame-hdr -m elf_x86_64 --hash-style=both -shared -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o -L/opt/site/lib -L/opt/site/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7 -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/opt/site/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o
    building '_Cryptography_cffi_26cb75b8x62b488b1' extension
    cc -O2 -fPIC -Wimplicit -I/opt/site/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o
    cc -shared -L/opt/site/lib -v build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
    Using built-in specs.
    COLLECT_GCC=cc
    COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
    Target: x86_64-linux-gnu
    Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
    Thread model: posix
    gcc version 4.7.2 (Debian 4.7.2-5)
    COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/
    LIBRARY_PATH=/opt/site/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/opt/site/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../:/lib/:/usr/lib/
    COLLECT_GCC_OPTIONS='-shared' '-L/opt/site/lib' '-v' '-o' 'build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so' '-mtune=generic' '-march=x86-64'
     /usr/lib/gcc/x86_64-linux-gnu/4.7/collect2 --sysroot=/ --build-id --no-add-needed --eh-frame-hdr -m elf_x86_64 --hash-style=both -shared -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o -L/opt/site/lib -L/opt/site/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7 -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/opt/site/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o
  Running setup.py install for pyOpenSSL
Successfully installed cryptography-0.8.1 enum34-1.0.4 pyOpenSSL-0.14 pyasn1-0.1.7 six-1.9.0
# /opt/site/bin/pypy -c 'import OpenSSL.SSL ; print("{:x}".format(OpenSSL.SSL.OPENSSL_VERSION_NUMBER))'
Traceback (most recent call last):
  File "<builtin>/app_main.py", line 75, in run_toplevel
  File "<builtin>/app_main.py", line 588, in run_it
  File "<string>", line 1, in <module>
  File "/opt/site/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/opt/site/site-packages/OpenSSL/rand.py", line 11, in <module>
    from OpenSSL._util import (
  File "/opt/site/site-packages/OpenSSL/_util.py", line 4, in <module>
    binding = Binding()
  File "/opt/site/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 113, in __init__
    self._ensure_ffi_initialized()
  File "/opt/site/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 123, in _ensure_ffi_initialized
    cls._modules,
  File "/opt/site/site-packages/cryptography/hazmat/bindings/utils.py", line 31, in load_library_for_binding
    lib = ffi.verifier.load_library()
  File ".../pypy-2.5.1-linux_x86_64-portable/lib_pypy/cffi/verifier.py", line 97, in load_library
    return self._load_library()
  File ".../pypy-2.5.1-linux_x86_64-portable/lib_pypy/cffi/verifier.py", line 207, in _load_library
    return self._vengine.load_library()
  File ".../pypy-2.5.1-linux_x86_64-portable/lib_pypy/cffi/vengine_gen.py", line 66, in load_library
    module = backend.load_library(filename, flags)
OSError: Cannot load library /opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so: /opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so: symbol EC_GFp_nistp521_method, version OPENSSL_1.0.1 not defined in file libcrypto.so.1.0.0 with link time reference
posita commented 9 years ago

Ah-hah! I figured out what I did to get it to work:

# rm -fr /opt/site/* ~/.cache ~/.pip && .../pypy-2.5.1-linux_x86_64-portable/bin/virtualenv-pypy /opt/site
New pypy executable in /opt/site/bin/pypy
Installing setuptools, pip...done.
# /opt/site/bin/pip install pyOpenSSL
...
Collecting cryptography>=0.2.1 (from pyOpenSSL)
  Downloading cryptography-0.8.1.tar.gz (268kB)
    100% |################################| 270kB 921kB/s
...
Successfully installed cryptography-0.8.1 enum34-1.0.4 pyOpenSSL-0.14 pyasn1-0.1.7 six-1.9.0
# /opt/site/bin/pypy -c 'import OpenSSL.SSL ; print("{:x}".format(OpenSSL.SSL.OPENSSL_VERSION_NUMBER))'
...
OSError: Cannot load library /opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so: /opt/site/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so: symbol EC_GFp_nistp521_method, version OPENSSL_1.0.1 not defined in file libcrypto.so.1.0.0 with link time reference
# /opt/site/bin/pip install https://pypi.python.org/packages/source/c/cryptography/cryptography-0.2.1.tar.gz
...
Successfully installed cryptography-0.2.1
# /opt/site/bin/pypy -c 'import OpenSSL.SSL ; print("{:x}".format(OpenSSL.SSL.OPENSSL_VERSION_NUMBER))'
1000105f

cryptography-0.8.1 is installed as the default dependency for pyOpenSSL, but forcing a downgrade to cryptography-0.2.1 seems to do the trick? :suspect:

posita commented 9 years ago

Closing as invalid. If anything, this is probably a pyOpenSSL or cryptography issue. I will reopen if appropriate.

posita commented 9 years ago

Reopening. I still think there's an issue here. It doesn't look like any of the libssl/libcrypto headers are provided with pypy-portable. This means that even though the compile phase includes -I[VIRTUALENV]/include, there aren't any OpenSSL headers there, so it will find the system ones (if they're around). This likely causes linking issues, since any cffi module will be built against the system headers.

UPDATE: In all likelihood, headers should be provided for all third party bundled libraries (expat, gdbm, sqlite3, etc.), not just OpenSSL. It is also likely that one will want to have the customary .so symlinks in [VIRTUALENV]/lib as well (e.g., libssl.so -> libssl.so.1.0.0).

posita commented 9 years ago

https://github.com/squeaky-pl/portable-pypy/blob/docker/build_deps#L15 As you can see here this version of portable pypy is built against version 1.0.2 of openssl ...

@squeaky-pl, does that mean builds are made from the docker branch? (FYI, I filed #10 to track missing build instructions.)

squeaky-pl commented 9 years ago

You are totally right about headers. Possibly also I should slightly patch distutils to automatically add those -L and -I switches and not use system-wide libraries and headers for libs that are bundled. I am gonna provide corrected build next week. I've switched to docker lately and 2.5.1 indeed was built from this branch. In the meantime I made a test-build that should let you proceed. It has all the header files in include manually copied and also missing symlinks in lib to build pyOpenSSL. After setting CFLAGS and LDFLAGS accordingly I was able to execute your test without linking errors.

https://bitbucket.org/squeaky/portable-pypy/downloads/pypy-2.5.1-issue9-linux_x86_64-portable.tar.gz

squeaky-pl commented 9 years ago

You will also want to store RPATH (-Wl,-rpath in LDFLAGS) in the resulting .sos. so the dynamic linker can resolve libraries during loading. Like this

CFLAGS=-I/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/include LDFLAGS="-L/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib  -Wl,-rpath,/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib" test-env/bin/pip install cryptography
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting cryptography
  Using cached cryptography-0.8.2.tar.gz
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in ./test-env/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in ./test-env/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./test-env/site-packages (from cryptography)
Requirement already satisfied (use --upgrade to upgrade): enum34 in ./test-env/site-packages (from cryptography)
Installing collected packages: cryptography
  Running setup.py install for cryptography
    building '_Cryptography_cffi_f3e4673fx399b1113' extension
    cc -O2 -fPIC -Wimplicit -I/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/include -I/home/squeaky/workspace/portable-pypy/test-env/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o
    cc -shared -I/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/include -L/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib -Wl,-rpath,/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o -lssl -lcrypto -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
    building '_Cryptography_cffi_590da19fxffc7b1ce' extension
    cc -O2 -fPIC -Wimplicit -I/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/include -I/home/squeaky/workspace/portable-pypy/test-env/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o
    cc -shared -I/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/include -L/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib -Wl,-rpath,/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
    building '_Cryptography_cffi_26cb75b8x62b488b1' extension
    cc -O2 -fPIC -Wimplicit -I/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/include -I/home/squeaky/workspace/portable-pypy/test-env/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o
    cc -shared -I/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/include -L/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib -Wl,-rpath,/home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
Successfully installed cryptography-0.8.2

To get something like this

find test-env/site-packages/ -name "*.so" | xargs ldd
test-env/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so:
        linux-vdso.so.1 =>  (0x00007ffe77f71000)
        libssl.so.1.0.0 => /home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib/libssl.so.1.0.0 (0x00007f5b1c755000)
        libcrypto.so.1.0.0 => /home/squeaky/workspace/portable-pypy/pypy-2.5.1-linux_x86_64-portable/lib/libcrypto.so.1.0.0 (0x00007f5b1c302000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5b1bf2b000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5b1bd27000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5b1cc11000)
test-env/site-packages/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so:
        linux-vdso.so.1 =>  (0x00007ffcd71f5000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fc3dff16000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc3e04f0000)
test-env/site-packages/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so:
        linux-vdso.so.1 =>  (0x00007ffe639ef000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fcd991fa000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcd997d4000)
posita commented 9 years ago

Your custom build works when referencing the virtual env as well. :clap:

Possibly also I should slightly patch distutils to automatically add those -L and -I switches and not use system-wide libraries and headers for libs that are bundled.

I noticed -I.../include is already part of the compile phase (this works with virtual environments as well; see below), so if you're installing the third party headers into the same subtree, you probably don't need to set any additional CFLAGS. So perhaps any patch of distutils only needs to set the -L and -Wl,-rpath,... arguments for the linking phase?

Also works when referencing the virtual env subtree instead of original:

% .../pypy-2.5.1-linux_x86_64-portable/bin/virtualenv-pypy ~/testenv
New pypy executable in /home/vagrant/testenv/bin/pypy
Installing setuptools, pip...done.
% LDFLAGS="-L${HOME}/testenv/lib -Wl,-rpath,${HOME}/testenv/lib" ~/testenv/bin/pip install cryptography
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting cryptography
  Using cached cryptography-0.8.2.tar.gz
Collecting pyasn1 (from cryptography)
  Using cached pyasn1-0.1.7.tar.gz
Collecting six>=1.4.1 (from cryptography)
  Using cached six-1.9.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./testenv/site-packages (from cryptography)
Collecting enum34 (from cryptography)
  Using cached enum34-1.0.4.tar.gz
Installing collected packages: enum34, six, pyasn1, cryptography
  Running setup.py install for enum34

  Running setup.py install for pyasn1
  Running setup.py install for cryptography
    building '_Cryptography_cffi_f3e4673fx399b1113' extension
    cc -O2 -fPIC -Wimplicit -I/home/vagrant/testenv/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o
    cc -shared -L/home/vagrant/testenv/lib -Wl,-rpath,/home/vagrant/testenv/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_f3e4673fx399b1113.o -lssl -lcrypto -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
    building '_Cryptography_cffi_590da19fxffc7b1ce' extension
    cc -O2 -fPIC -Wimplicit -I/home/vagrant/testenv/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o
    cc -shared -L/home/vagrant/testenv/lib -Wl,-rpath,/home/vagrant/testenv/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_590da19fxffc7b1ce.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
    building '_Cryptography_cffi_26cb75b8x62b488b1' extension
    cc -O2 -fPIC -Wimplicit -I/home/vagrant/testenv/include -c src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o
    cc -shared -L/home/vagrant/testenv/lib -Wl,-rpath,/home/vagrant/testenv/lib build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_26cb75b8x62b488b1.o -o build/lib.linux-x86_64-2.7/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
Successfully installed cryptography-0.8.2 enum34-1.0.4 pyasn1-0.1.7 six-1.9.0
% ~/testenv/bin/pypy -c 'from cryptography.hazmat.bindings.openssl.binding import Binding ; b = Binding() ; print("{:x}".format(b.lib.OPENSSL_VERSION_NUMBER))'
1000201f
% for i in $( find ~/testenv/site-packages/cryptography -name \*.so ) ; do echo "${i}" ; ldd "${i}" ; done
/home/vagrant/testenv/site-packages/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
        linux-vdso.so.1 =>  (0x00007ffccdd84000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f059f463000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f059f9f8000)
/home/vagrant/testenv/site-packages/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
        linux-vdso.so.1 =>  (0x00007ffc09f37000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fad3f639000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fad3fbce000)
/home/vagrant/testenv/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
        linux-vdso.so.1 =>  (0x00007fffd59fd000)
        libssl.so.1.0.0 => /home/vagrant/testenv/lib/libssl.so.1.0.0 (0x00007f6228d26000)
        libcrypto.so.1.0.0 => /home/vagrant/testenv/lib/libcrypto.so.1.0.0 (0x00007f62288d4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6228541000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f622833d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f62291e7000)
squeaky-pl commented 9 years ago

Hi, I built proper builds with patched distutils that installs cryptography without modifying ENV vars, can you try it?

https://bitbucket.org/squeaky/portable-pypy/downloads/pypy-2.5.1-1-linux_x86_64-portable.tar.bz2

posita commented 9 years ago

WORKSFORME Nice work! :+1:

$ rm -fr ~/.cache ~/testenv
$ .../pypy-2.5.1-linux_x86_64-portable/bin/virtualenv-pypy ~/testenv
New pypy executable in /home/vagrant/testenv/bin/pypy
Installing setuptools, pip...done.
$ ~/testenv/bin/pip install cryptography
Collecting cryptography
  Downloading cryptography-0.8.2.tar.gz (268kB)
    100% |████████████████████████████████| 270kB 958kB/s
Collecting pyasn1 (from cryptography)
  Downloading pyasn1-0.1.7.tar.gz (68kB)
    100% |████████████████████████████████| 69kB 2.6MB/s
Collecting six>=1.4.1 (from cryptography)
  Downloading six-1.9.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./testenv/site-packages (from cryptography)
Collecting enum34 (from cryptography)
  Downloading enum34-1.0.4.tar.gz
Installing collected packages: pyasn1, six, enum34, cryptography
  Running setup.py install for pyasn1
  Running setup.py install for enum34
  Running setup.py install for cryptography
Successfully installed cryptography-0.8.2 enum34-1.0.4 pyasn1-0.1.7 six-1.9.0
$ ~/testenv/bin/pypy -c 'from cryptography.hazmat.bindings.openssl.binding import Binding ; b = Binding() ; print("{:x}".format(b.lib.OPENSSL_VERSION_NUMBER))'
1000201f
$ for i in $( find ~/testenv/site-packages/cryptography -name \*.so ) ; do echo "${i}" ; ldd "${i}" ; done
/home/vagrant/testenv/site-packages/cryptography/_Cryptography_cffi_590da19fxffc7b1ce.pypy-25.so
        linux-vdso.so.1 =>  (0x00007ffcc97b1000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3069190000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3069725000)
/home/vagrant/testenv/site-packages/cryptography/_Cryptography_cffi_26cb75b8x62b488b1.pypy-25.so
        linux-vdso.so.1 =>  (0x00007ffc3a1da000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3c31d8b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3c32320000)
/home/vagrant/testenv/site-packages/cryptography/_Cryptography_cffi_f3e4673fx399b1113.pypy-25.so
        linux-vdso.so.1 =>  (0x00007fffa01fa000)
        libssl.so.1.0.0 => /home/vagrant/testenv/lib/libssl.so.1.0.0 (0x00007f1759770000)
        libcrypto.so.1.0.0 => /home/vagrant/testenv/lib/libcrypto.so.1.0.0 (0x00007f175931e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1758f8b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1758d87000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1759c31000)
squeaky-pl commented 9 years ago

Thank you very much. I am closing this then and updating the links to these builds.