Open mortvif opened 3 years ago
Just looking up the error "generated with LTO version 6.0 instead of the expected 8.1" seems to be somewhat common. Seems like possibly an outdated gcc version?
It do seems to be a problem with gcc versions. Aparently, the python version used within the conda environment has been compiled with gcc version 7.3 wheras the default gcc version on Ubuntu 20.04 is 9.3. Unfortunately, 7.3 version is not available anymore on Ubuntu 20.04.
Therefore, I am still unable to use pynode and the conda environment at the same time. I have avoided this problem by using the "native" (non-conda) python version which is compiled using gcc-v9.3. In this case, I can install and use Pynode without any issues.
I just succeeded in using PyNode with Miniconda on Ubuntu 18.04. Just to share how I finally make it:
conda create -n py38 python=3.8
, conda activate py38
pip install gyp-next find_libpython
find_libpython
or python -m find_libpython
. Suppose it be /xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0
.PYTHON_SHARED=/xxx/miniconda3/lib PY_LIBS="$(python ../build_ldflags.py) -L$PYTHON_SHARED -Wl,-rpath=$PYTHON_SHARED" npm install @fridgerator/pynode
undefined symbol: PyLong_AsLongLongAndOverflow
when importing some python module, just use pynode.dlOpen(path)
to link to this shared library again with path=/xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0
(i.e. the output of find_libpython
).At first, I encountered the lto1
error as well:
lto1: fatal error: bytecode stream in file ‘/home/sandra/anaconda3/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.a’ generated with LTO version 6.0 instead of the expected 8.1
I believe this is because the python installed and currently used on Conda is built by outdated GCC version as @fridgerator has mentioned.
Then I installed another python environment in Conda with a newer version:
conda create -n py38 python=3.8
(restart shell)
conda activate py38
Then you might find an error that -llibpython3.8
is not found:
/usr/bin/ld: cannot find -lpython3.8
This is because the python executable installed by conda is built by a static library like libpython3.8.a
rather than a shared library. If you check the dynamic library dependency of your python:
(py38) $ ldd `which python`
linux-vdso.so.1 (0x00007ffe03efd000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007faa187be000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007faa185bb000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007faa1821d000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007faa17ffe000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007faa17c0d000)
/lib64/ld-linux-x86-64.so.2 (0x00007faa18d5d000)
There is no such thing as libpython3.8.so
, that's why we can't find -lpython3.8
. What we need to do is to add the directory where libpython3.8.so
exists in the LDFLAGS, by adding -L<path_to_lib>
to environment variable PY_LIBS
leveraged by PyNode.
Fortunately, (1) conda still provides the dynamic library, and (2) there is a python package find_libpython
that could help us find the path.
(py38) $ pip install find_libpython
(py38) $ find_libpython
/xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0
Then let PY_LIBS
be the concatenation of the directory path and the output of python build_ldflags.py
:
(py38) $ PY_LIBS="$(python build_ldflags.py) -L/xxx/miniconda3/envs/py38/lib" npm install @fridgerator/pynode
Thus I installed PyNode successfully. However node.js still failed to find the shared library when it executing the PyNode addon, and we will get:
(py38) $ node
> const pynode = require('@fridgerator/pynode');
Uncaught:
Error: libpython3.8.so.1.0: cannot open shared object file: No such file or directory
at Object.Module._extensions..node (node:internal/modules/cjs/loader:1168:18)
at Module.load (node:internal/modules/cjs/loader:989:32)
at Function.Module._load (node:internal/modules/cjs/loader:829:14)
at Module.require (node:internal/modules/cjs/loader:1013:19)
at require (node:internal/modules/cjs/helpers:93:18) {
code: 'ERR_DLOPEN_FAILED'
}
(py38) $ ldd node_modules/@fridgerator/pynode/build/Release/PyNode.node
linux-vdso.so.1 (0x00007ffce1781000)
libpython3.8.so.1.0 => not found
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd675499000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd675110000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd674ef8000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd674b07000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd6758ba000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd674769000)
We need to add a -Wl,-rpath
option to tell the path to the directory like:
-Wl,-rpath=/xxx/miniconda3/envs/py38/lib
, which means actually we need to install PyNode like:
(py38) $ PYTHON_SHARED=/xxx/miniconda3/lib PY_LIBS="$(python ../build_ldflags.py) -L$PYTHON_SHARED -Wl,-rpath=$PYTHON_SHARED" npm install @fridgerator/pynode
And thus pynode would work well:
(py38) $ ldd node_modules/@fridgerator/pynode/build/Release/PyNode.node
linux-vdso.so.1 (0x00007ffca67dc000)
libpython3.8.so.1.0 => /xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0 (0x00007f2b384c2000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2b382be000)
libstdc++.so.6 => /mnt/asp_test/env/miniconda3/lib/libstdc++.so.6 (0x00007f2b38b23000)
libgcc_s.so.1 => /mnt/asp_test/env/miniconda3/lib/libgcc_s.so.1 (0x00007f2b38b0f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2b37ecd000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f2b37cca000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2b3792c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2b3770d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2b38a82000)
(py38) $ node
> const pynode = require('@fridgerator/pynode');
undefined
If you got an error like undefined symbol: PyLong_AsLongLongAndOverflow
when importing some python module, just use pynode.dlOpen(path)
to link to this shared library again with path=/xxx/miniconda3/envs/py38/lib/libpython3.8.so.1.0
(i.e. the output of find_libpython
).
@milliele Thanks for the write up! May I know the content of build_ldflags.py
you mentioned?
@milliele Thanks for the write up! May I know the content of
build_ldflags.py
you mentioned?
I didn't make any changes to the files. It's just this file in the repo: https://github.com/fridgerator/PyNode/blob/master/build_ldflags.py.
I'm trying to install PyNode 0.5.1 (last version available in npm) in my system using my usual anaconda environment to execute the python code. The versions currently used in my system are as follows:
However, I am facing an error while compiling the package. It seems that some packages are expected to be built with a different toolchain version (see attached log). Do you have any idea on how to solve this issue?
Thanks!