Closed facusapienza21 closed 1 year ago
Running the code leads to two segments, first some kind of summary, then a python traceback.
You can find the C code in this temporary file: /tmp/theano_compilation_error_pw8rhar_
library inux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: is not found.
Looking in the tmp file, I find C (C++?) code looking like this that ends with a more detailed error message:
===============================
1 #include <Python.h>
2 #include <iostream>
3 #include "theano_mod_helper.h"
4 #include <math.h>
5 #include <numpy/arrayobject.h>
...
586 return m;
587 }
588
===============================
Problem occurred during compilation with the command line below:
/srv/conda/envs/notebook/bin/g++ -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -Wno-c++11-narrowing -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -I/srv/conda/envs/notebook/lib/python3.9/site-packages/numpy/core/include -I/srv/conda/envs/notebook/include/python3.9 -I/srv/conda/envs/notebook/lib/python3.9/site-packages/theano/link/c/c_code -L/srv/conda/envs/notebook/lib -fvisibility=hidden -o /home/jovyan/.theano/compiledir_Linux-5.4-amzn2.x86_64-x86_64-with-glibc2.31-x86_64-3.9.9-64/tmpwnsmohb_/m31975b6dadb59d49af3f2b36405378825884299512ee0ed9081be33fa591d30d.so /home/jovyan/.theano/compiledir_Linux-5.4-amzn2.x86_64-x86_64-with-glibc2.31-x86_64-3.9.9-64/tmpwnsmohb_/mod.cpp -lpython3.9/srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /usr/lib64/libc_nonshared.a
collect2: error: ld returned 1 exit status
The failure relates to pymc3 relies on theano, and theano relies on compiling C++ code, and that C++ code relies on the linker ld
to have access to a relevant library that it fails to find. I'm not confident on how to solve this, but perhaps:
apt-get install
commandlibc6-dev
as described here and try doing that.Installing libc6-dev
didn't help. The failure is related to g++
compilation of code as invoked by pymc3-theano
. The failure is for the ld
program to find a specific file: /usr/lib64/libc_nonshared.a
. This file is available at /usr/lib/x86_64-linux-gnu/libc_nonshared.a
, but I've also failed to influence to make it detected. The issue is very similar to whats reported here by someone more knowledgeable about C++: https://github.com/awslabs/aws-lambda-cpp/issues/96
@abbyazari @fperez - do you have ideas on how to go about this? I've been down trying to debug these situations in the past and typically fail.
Note that I couldn't make LD_LIBRARY_PATH
or LIBRARY_PATH
influence the ld
executable to find the libc_nonshared.a
file. I also don't understand why or what makes it look for the file in the wrong folder. I think the LD_LIBRARY_PATH and such envs are relevant based on https://stackoverflow.com/a/4250666/2220152 and past experience. I'm not confident the ld
executable had those environment variables set though, because there is a lot of environment variable manipulation related to these ones... I ran into a rabbit hole about this in the past when sudo
was involved. Currently we are not using sudo
though - this is done as the jovyan non-sudo user.
I think I'm stuck at this point trying to figure this out =/
Thanks for the input and hard work, @consideRatio! These issues are a royal pain, and my most recent attempts at similar ones have not been super successful... I used to do a lot of this kind of work but most of my knowledge is a bit old and pre-conda days. I think we're experiencing some nasty interactions between system compilers/configuration and conda tools...
I'll try to make a local docker build where I can run these kinds of manual linux experiments purely by hand, but I would suggest pinging folks on the 2i2c slack - perhaps someone else on the team has experienced similar problems and can make a suggestion...
@consideRatio thank you for working on this and tracking it down! I have not run into this type of issue myself yet, I would second @fperez and see if the 2i2c folks have seen this before, I'll do some digging myself too.
Small update @consideRatio / @facusapienza21. @fperez has created a new environment that is working with pymc3. To access the directions are here:
create in your $HOME directory on the hub a file called .condarc, in .condarc put: envs_dirs: - ~/shared/envs - ~/shared-readwrite/envs
When running pymc3 notebooks then change the kernel to be the pymc3 kernel. More updates pending from @fperez on the exact nature of the pymc3 environment.
The embarrassing part is that I don't recall what I did! We can check which packages are installed in that env and do a comparison, as I think the solution is in one of the installed tools...
I still think there was a problem I wasn't quite able to solve with local compilation, and perhaps that's why I thought I hadn't quite fixed the pymc issue. But it seems for now that env works for pymc, so we can use that and keep iterating on this question.
I could reproduce successfully using pymc3 with the example @facusapienza21 provides at the top post of this issue. Comparing the included content, I don't see much of importance be changed. I think these could be relatively interesting parts...
-libclang 11.1.0 default_ha53f305_1 conda-forge
+libclang 13.0.0 default_hc23dcda_0 conda-forge
-abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge
-arrow-cpp 6.0.1 py39h01fd06f_8_cpu conda-forge
-compilers 1.3.0 ha770c72_0 conda-forge
-cython 0.29.27 py39he80948d_0 conda-forge
gcc 9.4.0 h192d537_4 conda-forge
gcc_impl_linux-64 9.4.0 h03d3576_12 conda-forge
gcc_linux-64 9.4.0 h391b98a_4 conda-forge
mkl 2021.4.0 h8d4b97c_729 conda-forge
mkl-service 2.4.0 py39h7e14d7c_0 conda-forge
I know that I've added compilers
before in an attempt to help make regreg
installable (Related: https://github.com/pangeo-data/jupyter-earth/issues/66).
Perhaps uninstalling compilers
is relevant? Perhaps trying to update to a modern libclang
which probably has been installed by another dependency though - like arrow-cpp
or abseil-cpp
I suspect.
Closing as resolved by relying on the functional environment with pymc3 in shared/envs/pymc3 on the shared network file system.
Following #99, even when both PyMC3 and Theano-PyMC seem to be installed, when trying to execute code based on PyMC I always see the same error. Here there is a minimum example extracted from Getting started with PyMC3:
A similar error was reported here, where they mention something about the installation of Theano. However, the version of theano installed in the Hub seems to be the rigth one.
@abbyazari @consideRatio