search

nbigaouette / hygeia

Python toolchain manager 🐍
Apache License 2.0
68 stars 3 forks source link

Python <3.3.0 has problem compiling with --enable-shared #122

Open nbigaouette opened 4 years ago

nbigaouette commented 4 years ago

If --enable-shared is used to compile Python <3.3.0, linking fails but is hard to see directly in pycors as it simply hangs:

⠠ [4/15] Make: gcc -bundle -bundle_loader python.exe -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt...

pstree shows the calling hierarchy:

[...]
 | | \-+= 23880 nbigaouette target/debug/pycors install =3.2.6
 | |   \-+- 29709 nbigaouette /Library/Developer/CommandLineTools/usr/bin/make
 | |     \-+- 29962 nbigaouette /bin/sh -c case "$MAKEFLAGS" in     *\ -s*|s*) quiet="-q";;     *) quiet="";; esac; DYLD_LIBRARY_PATH=`pwd`: CC='gcc' LDSHARED='gcc -bundle -bundle_loader python.exe -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib' OPT='-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes' \011./python.exe -E ./setup.py $quiet build
 | |       \-+- 29964 nbigaouette ./python.exe -E ./setup.py build
 | |         \-+- 30035 nbigaouette /Library/Developer/CommandLineTools/usr/bin/clang -bundle -bundle_loader python.exe -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/openssl/include -I/opt/X11/include build/temp.macosx-10.15-x86_64-3.2/Users/nbigaouette/.pycors/cache/extracted/Python-3.2.6/Modules/_pickle.o -L/usr/local/opt/openssl/lib -L/usr/local/lib -o build/lib.macosx-10.15-x86_64-3.2/_pickle.so
 | |           \--- 30036 nbigaouette /Library/Developer/CommandLineTools/usr/bin/ld -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -dynamic -arch x86_64 -bundle -bundle_loader python.exe -macosx_version_min 10.15.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o build/lib.macosx-10.15-x86_64-3.2/_pickle.so -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -L/usr/local/lib build/temp.macosx-10.15-x86_64-3.2/Users/nbigaouette/.pycors/cache/extracted/Python-3.2.6/Modules/_pickle.o -L/usr/local/lib -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.osx.a

Instead, running this manually exposes the linking problem (on macOS Catalina):

> cd ~/.pycors/cache/extracted/Python-3.2.6
> ./configure --prefix ${HOME}/.pycors/installed/3.2.6 --enable-shared
[...]
> make
[...]
gcc  -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes     -I. -IInclude -I./Include    -DPy_BUILD_CORE  -c ./Modules/xxsubtype.c
-o Modules/xxsubtype.o
gcc -c  -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes     -I. -IInclude -I./Include    -DPy_BUILD_CORE \
              -DSVNVERSION="\"`LC_ALL=C svnversion .`\"" \
              -DHGVERSION="\"`LC_ALL=C `\"" \                                                                                                   -DHGTAG="\"`LC_ALL=C `\"" \                                                                                                       -DHGBRANCH="\"`LC_ALL=C `\"" \                                                                                                    -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
rm -f libpython3.2m.a
ar rc libpython3.2m.a Modules/getbuildinfo.o
ar rc libpython3.2m.a Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/
bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/tokenizer.o
ar rc libpython3.2m.a Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Obj
ects/bytesobject.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o O
bjects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Obje
cts/funcobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/memoryobject.o Object
s/methodobject.o Objects/moduleobject.o Objects/object.o Objects/obmalloc.o Objects/capsule.o Objects/rangeobject.o Objects/setobj
ect.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicode
ctype.o Objects/weakrefobject.o
ar rc libpython3.2m.a Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/bltinmodule.o Python/ceval.o Python
/compile.o Python/codecs.o Python/dynamic_annotations.o Python/errors.o Python/frozen.o Python/frozenmain.o Python/future.o Python
/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o P
ython/importdl.o Python/marshal.o Python/modsupport.o Python/mystrtoul.o Python/mysnprintf.o Python/peephole.o Python/pyarena.o Py
thon/pyctype.o Python/pyfpe.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/random.o Python/structmem
ber.o Python/symtable.o Python/sysmodule.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/dtoa.o Py
thon/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o   Python/thread.o
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libpython3.2m.a(dynamic_annotations.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libpython3.2m.a(pymath.o) has no symbols
ar rc libpython3.2m.a Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libpython3.2m.a(dynamic_annotations.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libpython3.2m.a(pymath.o) has no symbols
ar rc libpython3.2m.a Modules/_threadmodule.o  Modules/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmo
dule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/operator.o  Modules/_coll
ectionsmodule.o  Modules/itertoolsmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules
/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/zipimport.o  Modules/symtablemodule.o  Modules/xxsubtype.o
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libpython3.2m.a(dynamic_annotations.o) has no symbols        [536/14326]
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libpython3.2m.a(pymath.o) has no symbols
ranlib libpython3.2m.a
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libpython3.2m.a(dynamic_annotations.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libpython3.2m.a(pymath.o) has no symbols
gcc -dynamiclib -Wl,-single_module   -undefined dynamic_lookup -Wl,-install_name,/Users/nbigaouette/.pycors/installed/3.2.6/lib/li
bpython3.2m.dylib -Wl,-compatibility_version,3.2 -Wl,-current_version,3.2 -o libpython3.2m.dylib Modules/getbuildinfo.o Parser/acc
eler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Pa
rser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/b
oolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/cellobject.o Objects/classobject.o Obj
ects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Obje
cts/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/iterobject.o Objects/listobject.o Object
s/longobject.o Objects/dictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/object.o Objects/
obmalloc.o Objects/capsule.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobje
ct.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-
ast.o Python/asdl.o Python/ast.o Python/bltinmodule.o Python/ceval.o Python/compile.o Python/codecs.o Python/dynamic_annotations.o
 Python/errors.o Python/frozen.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o P
ython/getplatform.o Python/getversion.o Python/graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o P
ython/mystrtoul.o Python/mysnprintf.o Python/peephole.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pymath.o Python/py
state.o Python/pythonrun.o Python/pytime.o Python/random.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/trace
back.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynl
oad_shlib.o   Python/thread.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o  Modules/_threadmodule.o  Modul
es/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Mod
ules/_weakref.o  Modules/_functoolsmodule.o  Modules/operator.o  Modules/_collectionsmodule.o  Modules/itertoolsmodule.o  Modules/
_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Mod
ules/stringio.o  Modules/zipimport.o  Modules/symtablemodule.o  Modules/xxsubtype.o -ldl  -framework CoreFoundation   ; \

gcc   -framework CoreFoundation -o python.exe Modules/python.o -L. -lpython3.2m -ldl  -framework CoreFoundation
running build
running build_ext
building '_struct' extension
gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -IInclude -I/Users/nbigaouette/.pycors/installed/3.2.6/include -I. -I./Inclu
de -I/usr/local/include -I/Users/nbigaouette/.pycors/cache/extracted/Python-3.2.6 -c /Users/nbigaouette/.pycors/cache/extracted/Py
thon-3.2.6/Modules/_struct.c -o build/temp.macosx-10.15-x86_64-3.2/Users/nbigaouette/.pycors/cache/extracted/Python-3.2.6/Modules/
_struct.o
gcc -bundle -bundle_loader python.exe build/temp.macosx-10.15-x86_64-3.2/Users/nbigaouette/.pycors/cache/extracted/Python-3.2.6/Mo
dules/_struct.o -L/Users/nbigaouette/.pycors/installed/3.2.6/lib -L/usr/local/lib -o build/lib.macosx-10.15-x86_64-3.2/_struct.so
Undefined symbols for architecture x86_64:
  "_PyArg_UnpackTuple", referenced from:
      _unpack in _struct.o
  "_PyBool_FromLong", referenced from:
      _nu_bool in _struct.o
      _bu_bool in _struct.o
  "_PyBuffer_Release", referenced from:
      _unpack in _struct.o
      _s_unpack in _struct.o
      _s_unpack_from in _struct.o
  "_PyByteArray_Type", referenced from:
      _s_pack_internal in _struct.o
  "_PyBytes_AsString", referenced from:
      _np_char in _struct.o
  "_PyBytes_FromStringAndSize", referenced from:
      _pack in _struct.o
      _s_pack in _struct.o
      _s_unpack_internal in _struct.o
      _nu_char in _struct.o
  "_PyBytes_Size", referenced from:
      _np_char in _struct.o
  "_PyDict_Clear", referenced from:
      _cache_struct in _struct.o
  "_PyDict_GetItem", referenced from:
      _cache_struct in _struct.o
  "_PyDict_New", referenced from:
      _cache_struct in _struct.o
  "_PyDict_SetItem", referenced from:
      _cache_struct in _struct.o
  "_PyDict_Size", referenced from:
      _cache_struct in _struct.o
  "_PyErr_Clear", referenced from:
      _cache_struct in _struct.o
  "_PyErr_ExceptionMatches", referenced from:
      _s_pack_internal in _struct.o
      _np_byte in _struct.o
      _np_ubyte in _struct.o
      _np_short in _struct.o
      _np_ushort in _struct.o
      _np_int in _struct.o
      _np_uint in _struct.o
      ...
  "_PyErr_Format", referenced from:
      _pack in _struct.o
      _unpack in _struct.o
      _s_pack in _struct.o
      _s_pack_into in _struct.o
      _s_unpack in _struct.o
      _s_unpack_from in _struct.o
      _s_init in _struct.o
      ...
  "_PyErr_NewException", referenced from:
      _PyInit__struct in _struct.o
  "_PyErr_NoMemory", referenced from:
      _s_init in _struct.o
  "_PyErr_Occurred", referenced from:
      _s_pack_into in _struct.o
      _np_byte in _struct.o
      _np_ubyte in _struct.o
      _np_short in _struct.o
      _np_ushort in _struct.o
      _np_int in _struct.o
      _np_uint in _struct.o
      ...
  "_PyErr_SetString", referenced from:
      _pack in _struct.o
      _pack_into in _struct.o
      _unpack_from in _struct.o
      _s_pack_internal in _struct.o
      _s_init in _struct.o
      _np_byte in _struct.o
      _np_ubyte in _struct.o                                                                                           [428/14326]
      ...
  "_PyExc_IndexError", referenced from:
      _s_pack_into in _struct.o
  "_PyExc_OverflowError", referenced from:
      _s_pack_internal in _struct.o
      _np_byte in _struct.o
      _np_ubyte in _struct.o
      _np_short in _struct.o
      _np_ushort in _struct.o
      _np_int in _struct.o
      _np_uint in _struct.o
      ...
  "_PyExc_TypeError", referenced from:
      _pack in _struct.o
      _pack_into in _struct.o
      _unpack_from in _struct.o
      _s_init in _struct.o
  "_PyFloat_AsDouble", referenced from:
      _np_float in _struct.o
      _np_double in _struct.o
      _lp_float in _struct.o
      _lp_double in _struct.o
      _bp_float in _struct.o
      _bp_double in _struct.o
  "_PyFloat_FromDouble", referenced from:
      _nu_float in _struct.o
      _nu_double in _struct.o
      _lu_float in _struct.o
      _lu_double in _struct.o
      _bu_float in _struct.o
      _bu_double in _struct.o
  "_PyLong_AsLong", referenced from:
      _np_byte in _struct.o
      _np_ubyte in _struct.o
      _np_short in _struct.o
      _np_ushort in _struct.o
[...]
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
building '_ctypes_test' extension
gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -IInclude -I/Users/nbigaouette/.pycors/installed/3.2.6/include -I. -I./Inclu
de -I/usr/local/include -I/Users/nbigaouette/.pycors/cache/extracted/Python-3.2.6 -c /Users/nbigaouette/.pycors/cache/extracted/Py
thon-3.2.6/Modules/_ctypes/_ctypes_test.c -o build/temp.macosx-10.15-x86_64-3.2/Users/nbigaouette/.pycors/cache/extracted/Python-3
.2.6/Modules/_ctypes/_ctypes_test.o
gcc -bundle -bundle_loader python.exe build/temp.macosx-10.15-x86_64-3.2/Users/nbigaouette/.pycors/cache/extracted/Python-3.2.6/Mo
dules/_ctypes/_ctypes_test.o -L/Users/nbigaouette/.pycors/installed/3.2.6/lib -L/usr/local/lib -o build/lib.macosx-10.15-x86_64-3.
2/_ctypes_test.so
Undefined symbols for architecture x86_64:
  "_PyArg_ParseTuple", referenced from:
      _py_func_si in _ctypes_test.o
  "_PyModule_Create2", referenced from:
      _PyInit__ctypes_test in _ctypes_test.o
  "__Py_NoneStruct", referenced from:
      _py_func_si in _ctypes_test.o
      _py_func in _ctypes_test.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
building 'array' extension
tkamenoko commented 4 years ago

--enable-shared may also cause problems with python>=3.3 .

package distributors like https://packages.debian.org/en/sid/libpython3-dev , they don't consider python's patch version, or may not provide the newest library.

I installed python=3.8.1 with hygeia , and I got error messages like this:

$ python3 python3: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory Error: Failed to execute command (exit code: Some(127)): python3

so I installed libpython=3.8 , but got libpython3.8.0 , so python3 -V showed version 3.8.0 .

nbigaouette commented 4 years ago

Which distro are you using? Debian, Ubuntu? And which version?

When compiling on Linux I use LD_RUN_PATH to encode the path to the shared libraries inside the built binary. This way python should pick up its own shared library when run. See: https://github.com/nbigaouette/hygeia/blob/a19d29927573cedcf616ac40e82dd5d59ae55e47/src/commands/install/unix.rs#L151-L155

It seems that if you have a shared library installed system wide it is picked up first.

Can you try to call python directly using LD_LIBRARY_PATH and see what happens? The binary should pick up its own library.

LD_LIBRARY_PATH=${HYGEIA_HOME}/installed/cpython/3.8.1/lib ${HYGEIA_HOME}/installed/cpython/3.8.1/bin/python -V

If that works I'll try to think of something...

nbigaouette commented 4 years ago

I'm having trouble reproducing your problem.

I'm using docker to get a debian sid (sid-20191224) environment. This one comes with Python 3.7.6:

hygeia@2d18f7390b85:~$ hygeia list
+--------+---------+---------------------+--------------------------------------------------+
| Active | Version | Installed by hygeia | Location                                         |
+--------+---------+---------------------+--------------------------------------------------+
|        |  3.7.6  |                     | /usr/bin                                         |
+--------+---------+---------------------+--------------------------------------------------+
hygeia@2d18f7390b85:~$ /usr/bin/python3 -V
Python 3.7.6

The binary is not dynamically link to libpython:

hygeia@2d18f7390b85:~$ ldd /usr/bin/python3
        linux-vdso.so.1 (0x00007ffdb396a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6c46eee000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6c46ecd000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6c46ec8000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f6c46ec3000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f6c46e96000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6c46e79000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6c46d32000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6c470b4000)
hygeia@2d18f7390b85:~$ /usr/bin/python3 -m sysconfig -c 'print(sysconfig.get_config_vars())' | grep -i shared
        BLDSHARED = "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions  -Wl,-z,relro"
        CCSHARED = "-fPIC"
        CFLAGSFORSHARED = "-fPIC"
        CONFIG_ARGS = "'--enable-shared' '--prefix=/usr' '--enable-ipv6' '--enable-loadable-sqlite-extensions' '--with-dbmliborder=bdb:gdbm' '--with-computed-gotos' '--without-ensurepip' '--with-system-expat' '--with-system-libmpdec' '--with-system-ffi' 'CC=x86_64-linux-gnu-gcc' 'CFLAGS=-g   -fstack-protector-strong -Wformat -Werror=format-security ' 'LDFLAGS= -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'"
        COVERAGE_INFO = "/build/python3.7-9TOSzB/python3.7-3.7.6/build-shared/coverage.info"
        COVERAGE_REPORT = "/build/python3.7-9TOSzB/python3.7-3.7.6/build-shared/lcov-report"
        DESTSHARED = "/usr/lib/python3.7/lib-dynload"
        INSTALL_SHARED = "/usr/bin/install -c -m 555"
        LDCXXSHARED = "x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions"
        LDSHARED = "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions  -Wl,-z,relro"
        LINKFORSHARED = "-Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions"
        Py_ENABLE_SHARED = "1"
        RUNSHARED = "LD_LIBRARY_PATH=/build/python3.7-9TOSzB/python3.7-3.7.6/build-shared"
        TESTPYTHON = "LD_LIBRARY_PATH=/build/python3.7-9TOSzB/python3.7-3.7.6/build-shared ./python"
        TESTRUNNER = "LD_LIBRARY_PATH=/build/python3.7-9TOSzB/python3.7-3.7.6/build-shared ./python ../Tools/scripts/run_tests.py"
        abs_builddir = "/build/python3.7-9TOSzB/python3.7-3.7.6/build-shared"
        abs_srcdir = "/build/python3.7-9TOSzB/python3.7-3.7.6/build-shared/.."

Installing 3.8.1 succeeds:

hygeia@2d18f7390b85:~$ hygeia list
+--------+---------+---------------------+--------------------------------------------------+
| Active | Version | Installed by hygeia | Location                                         |
+--------+---------+---------------------+--------------------------------------------------+
|        |  3.8.1  |          ✓          | /home/hygeia/.hygeia/installed/cpython/3.8.1/bin |
+--------+---------+---------------------+--------------------------------------------------+
|        |  3.7.6  |                     | /usr/bin                                         |
+--------+---------+---------------------+--------------------------------------------------+
hygeia@2d18f7390b85:~$ ldd /home/hygeia/.hygeia/installed/cpython/3.8.1/bin/python
        linux-vdso.so.1 (0x00007ffcb74e1000)
        libpython3.8.so.1.0 => /home/hygeia/.hygeia/installed/cpython/3.8.1/lib/libpython3.8.so.1.0 (0x00007fa2dda23000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa2dd85f000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa2dd83e000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa2dd839000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fa2dd834000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa2dd6ef000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa2ddda7000)
hygeia@2d18f7390b85:~$ /home/hygeia/.hygeia/installed/cpython/3.8.1/bin/python -m sysconfig -c 'print(sysconfig.get_config_vars())' | grep -i shared
        BLDSHARED = "gcc -pthread -shared"
        CCSHARED = "-fPIC"
        CFLAGSFORSHARED = "-fPIC"
        CONFIG_ARGS = "'--prefix' '/home/hygeia/.hygeia/installed/cpython/3.8.1' '--enable-shared' 'CFLAGS=' 'LDFLAGS=' 'CPPFLAGS='"
        DESTSHARED = "/home/hygeia/.hygeia/installed/cpython/3.8.1/lib/python3.8/lib-dynload"
        INSTALL_SHARED = "/usr/bin/install -c -m 755"
        LDCXXSHARED = "g++ -pthread -shared"
        LDSHARED = "gcc -pthread -shared"
        LINKFORSHARED = "-Xlinker -export-dynamic"
        Py_ENABLE_SHARED = "1"
        RUNSHARED = "LD_LIBRARY_PATH=/home/hygeia/.hygeia/cache/extracted/Python-3.8.1"
hygeia@2d18f7390b85:~$ /home/hygeia/.hygeia/installed/cpython/3.8.1/bin/python -V
Python 3.8.1
nbigaouette commented 4 years ago

Running python3 -m sysconfig -c 'print(sysconfig.get_config_vars())' might reveal what is wrong...

nbigaouette commented 4 years ago

http://blog.dscpl.com.au/2015/06/installing-custom-python-version-into.html explains why distro provided python is statically linked but still include a shared lib; TL;DR the toolchain is compiled twice, once with configure --enable-shared and once with make altbininstall... 🤔

nbigaouette commented 4 years ago

There two issues here:

  1. I failed to compile Python <3.3.0 using --enable-shared so hygeia simply doesn't use --enable-shared for <3.3.0. I'm not sure if this is "fixable", those versions might just be too old for compiling on more modern system... This issue is about keeping track of that problem.
  2. After installing 3.8.1 using hygeia, your setup does not find the shared library and fails to load. I'd say it's a bug in how python is being built on your system as the path to the shared lib should be added to the binary. That's a bug that needs to be fixed.
nbigaouette commented 4 years ago

I've opened #139 to track the separate issue.