Error on python installation

leejayyoon commented 6 years ago

Hi,

It seems my installation process finishes without problem, nonetheless, the simple import is spitting errors. I tested just installing c++ as well (cmake without -DPYTHON=which python ). Both cases, generated c++ examples seem to be running Okay, and thus I am suspecting somewhere in the library linking process is not working properly, but I am not too sure where ... I also didn't have problem with other server following similar procedures. The other one was Red Hat Linux, and it had lib, bins in more standard place: /usr/local/ .. whereas this computer has common things on /share/apps/. On last note, I don't have sudo and I made everything local. (This might be related to other open issues: #649 and #685 )

So to jump to the symptom, I am getting error when, I do, python -c "import dynet" This is the error log:

jisuk1@dual-970-0-7:~$ python -c 'import dynet as dy; print dy.__version__; pc = dy.ParameterCollection()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jisuk1/tools/dynet-base2/dynet/build/python/dynet.py", line 15, in <module>
    from _dynet import *
ImportError: /home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.so: undefined symbol: _ZN5dynet13TextFileSaver4saveERKNS_19ParameterCollectionERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE```

After describing the environment and configurations I made, at the end, I am putting the logs from make as well.

Environment: I am running on ubuntu 14.04, on GPU server with cudnn-5.1.5, cuda-8.0, gcc version 5.4.0

What I did I ran the following: cmake .. -DEIGEN3_INCLUDE_DIR=../../eigen -DPYTHON=which python-DBACKEND=cuda -DCUDNN_ROOT=/share/apps/cuda-8.0 and the rest is the same with the tutorial, i.e.,

make -j 32
python ../../setup.py build --build-dir=.. --skip-build install --user

My cudnn is on non-conventional location, so I specified the cudnn root. I tried multiple installation, which got all the same errors when importing, and this was the final form. While re-installing as I wanted to overcome the error,

I tried removing build directory completely and rerunning ...
I tried removing whole dynet repo and rerunning ...
I tried removing .local/lib/(some dynet .egg) & temporary location of _dynet.so
specified -DCUDNN_ROOT=/share/apps/cuda-8.0. None of these really worked except when I did the plain c++ installation. (Actually, plain c++ implementation's ./examples/xor works as well, but not python) So I am suspecting something was wrong on linking the library ... but I couldn't really get what is going wrong on the process.

PATH, LD_LIBRARY_PATH

On top of including the library file on the path, I also tried to include $PREFIX/.local/bin on PATH $PREFIX/.local/lib on LD_LIBRARY_PATH as I did 'local install'. I also tried putting dynet library path to the front ... but it didn't work regardless.

also have done export PYTHONPATH=$PREFIX/tools/dynet-base2/dynet/build/python:$PYTHONPATH following @pmichel31415 's instruction.

This is on my .bashrc:

export LD_LIBRARY_PATH=$PREFIX/tools/dynet-base2/dynet/build/dynet/:$LD_LIBRARY_PATH:$PREFIX/.local/lib:$PREFIX/local/lib/:$PREFIX/tools/protobuf/src/.libs
export DYLD_LIBRARY_PATH=$PREFIX/tools/dynet-base2/dynet/build/dynet/:$DYLD_LIBRARY_PATH
export PATH=$PATH:$PREFIX/local/bin/:$PREFIX/.local/bin

Here are the specified logs that I had during cmake, make, and python installation. cmake log

cmake .. -DEIGEN3_INCLUDE_DIR=/home/jisuk1/tools/dynet-base2/eigen -DPYTHON=`which python` -DENABLE_CPP_EXAMPLES=ON -DCUDNN_ROOT=/share/apps/cuda-8.0

-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- BACKEND not specified, defaulting to eigen.
-- Eigen dir is /home/jisuk1/tools/dynet-base2/eigen
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found Cython version 0.25.1

CMAKE_INSTALL_PREFIX="/usr/local"
PROJECT_SOURCE_DIR="/home/jisuk1/tools/dynet-base2/dynet"
PROJECT_BINARY_DIR="/home/jisuk1/tools/dynet-base2/dynet/build"
LIBS="-lpthread"
EIGEN3_INCLUDE_DIR="/home/jisuk1/tools/dynet-base2/eigen"
MKL_LINK_DIRS=""
WITH_CUDA_BACKEND=""
CUDA_RT_FILES=""
CUDA_RT_DIRS=""
CUDA_CUBLAS_FILES=""
CUDA_CUBLAS_DIRS=""
MSVC=""
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CUDNN_ROOT

-- Build files have been written to: /home/jisuk1/tools/dynet-base2/dynet/build

Make log

Scanning dependencies of target dynet
[ 12%] Building CXX object dynet/CMakeFiles/dynet.dir/grad-check.cc.o
[ 13%] Building CXX object dynet/CMakeFiles/dynet.dir/aligned-mem-pool.cc.o
[ 29%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-conv.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/devices.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/deep-lstm.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/model.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-arith-unary.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-arith-const.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-arith-sum.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/gru.cc.o
[ 31%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-concat.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/dim.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-arith-cwise.cc.o
[ 31%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-flow.cc.o
[ 31%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-contract.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/hsm-builder.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/expr.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/lstm.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/init.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/dynet.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/globals.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-affinetransform.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/exec.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/dict.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/fast-lstm.cc.o
[ 30%] Building CXX object dynet/CMakeFiles/dynet.dir/io.cc.o
[ 32%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-hinge.cc.o
[ 33%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-linalg.cc.o
[ 34%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-logsumexp.cc.o
[ 35%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-losses.cc.o
[ 36%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-lstm.cc.o
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc: In member function ‘virtual dynet::Dim dynet::CwiseSum::dim_forward(const std::vector<dynet::Dim>&) const’:
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:23:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for(int i=0; i < min(xs[0].nd, xs[1].nd); i++){
                                          ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:27:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for(int i=0; i < max(xs[0].nd, xs[1].nd); i++){
                                          ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:28:34: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if(i < min(xs[0].nd, xs[1].nd)) dims.push_back(max(xs[0].d[i], xs[1].d[i]));
                                  ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:29:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     else if(i < xs[0].nd) dims.push_back(xs[0].d[i]);
                       ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc: In member function ‘virtual dynet::Dim dynet::CwiseMultiply::dim_forward(const std::vector<dynet::Dim>&) const’:
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:151:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for(int i=0; i < min(xs[0].nd, xs[1].nd); i++){
                                          ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:155:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for(int i=0; i < max(xs[0].nd, xs[1].nd); i++){
                                          ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:156:34: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if(i < min(xs[0].nd, xs[1].nd)) dims.push_back(max(xs[0].d[i], xs[1].d[i]));
                                  ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:157:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     else if(i < xs[0].nd) dims.push_back(xs[0].d[i]);
                       ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc: In member function ‘virtual dynet::Dim dynet::CwiseQuotient::dim_forward(const std::vector<dynet::Dim>&) const’:
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:278:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for(int i=0; i < min(xs[0].nd, xs[1].nd); i++){
                                          ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:282:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for(int i=0; i < max(xs[0].nd, xs[1].nd); i++){
                                          ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:283:34: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if(i < min(xs[0].nd, xs[1].nd)) dims.push_back(max(xs[0].d[i], xs[1].d[i]));
                                  ^
/home/jisuk1/tools/dynet-base2/dynet/dynet/nodes-arith-cwise.cc:284:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     else if(i < xs[0].nd) dims.push_back(xs[0].d[i]);
                       ^
[ 37%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-matrixmultiply.cc.o
[ 38%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-maxpooling2d.cc.o
[ 39%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-minmax.cc.o
[ 40%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-moments.cc.o
[ 41%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-normalization.cc.o
[ 42%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-norms.cc.o
[ 43%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-pickneglogsoftmax.cc.o
[ 45%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-similarities.cc.o
[ 45%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-random.cc.o
[ 45%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-select.cc.o
[ 47%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-softmaxes.cc.o
[ 47%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-to-device.cc.o
[ 48%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-trig.cc.o
[ 49%] Building CXX object dynet/CMakeFiles/dynet.dir/param-init.cc.o
[ 50%] Building CXX object dynet/CMakeFiles/dynet.dir/param-nodes.cc.o
[ 51%] Building CXX object dynet/CMakeFiles/dynet.dir/pretrain.cc.o
[ 52%] Building CXX object dynet/CMakeFiles/dynet.dir/rnn-state-machine.cc.o
[ 53%] Building CXX object dynet/CMakeFiles/dynet.dir/rnn.cc.o
[ 54%] Building CXX object dynet/CMakeFiles/dynet.dir/saxe-init.cc.o
[ 55%] Building CXX object dynet/CMakeFiles/dynet.dir/shadow-params.cc.o
[ 57%] Building CXX object dynet/CMakeFiles/dynet.dir/tensor.cc.o
[ 57%] Building CXX object dynet/CMakeFiles/dynet.dir/training.cc.o
[ 58%] Building CXX object dynet/CMakeFiles/dynet.dir/treelstm.cc.o
[ 59%] Building CXX object dynet/CMakeFiles/dynet.dir/weight-decay.cc.o
[ 60%] Linking CXX shared library libdynet.so
[ 60%] Built target dynet
Scanning dependencies of target pydynet_precopy
Scanning dependencies of target embed-cl
Scanning dependencies of target read-write
Scanning dependencies of target attention
Scanning dependencies of target poissoScanning dependencies of target tag-bilstm
Scanning dependencies of target 1_linear_regression
Scanning dependencies of target rnnlm-batch-nce
Scanning dependencies of target xor
Scanning dependencies of target 0_multiply
Scanning dependencies of target rnn-autobatch
[ 61%] Building CXX object examples/CMakeFiles/rnnlm-aevb.dir/variational-autoencoder/train_rnnlm-aevb.cc.o
[ 70%] Building CXX object examples/CMakeFiles/xor-batch.dir/batching/train_xor-batch.cc.o
[ 71%] Building CXX object tutorial/CMakeFiles/0_multiply.dir/0_multiply.cc.o
[ 76%] Building CXX object tutorial/CMakeFiles/1_linear_regression.dir/1_linear_regression.cc.o
[ 76%] Building CXX object examples/CMakeFiles/rnnlm-batch.dir/batching/train_rnnlm-batch.cc.o
[ 76%] Building CXX object examples/CMakeFiles/mnist.dir/mnist/train_mnist.cc.o
[ 79%] Building CXX object examples/CMakeFiles/imdb.dir/document-classification/train_imdb.cc.o
[ 79%] Building CXX object examples/CMakeFiles/encdec.dir/sequence-to-sequence/train_encdec.cc.o
[ 79%] Building CXX object examples/CMakeFiles/rnnlm.dir/rnnlm/train_rnnlm.cc.o
[ 79%] Building CXX object examples/CMakeFiles/rnn-autobatch.dir/autobatch/train_rnn-autobatch.cc.o
[ 79%] Building CXX object examples/CMakeFiles/xor.dir/xor/train_xor.cc.o
[ 79%] Building CXX object examples/CMakeFiles/rnnlm-batch-nce.dir/noise-contrastive-estimation/train_rnnlm-batch-nce.cc.o
[ 79%] Building CXX object examples/CMakeFiles/poisson-regression.dir/poisson-regression/train_poisson-regression.cc.o
[ 79%] Building CXX object examples/CMakeFiles/rnnlm-cfsm.dir/softmax-builders/train_rnnlm-cfsm.cc.o
[ 70%] Building CXX object examples/CMakeFiles/tag-bilstm.dir/tagger/train_tag-bilstm.cc.o
[ 76%] Building CXX object examples/CMakeFiles/attention.dir/sequence-to-sequence/train_attention.cc.o
[ 79%] Building CXX object examples/CMakeFiles/tok-embed.dir/word-embedding/train_tok-embed.cc.o
[ 79%] Building CXX object examples/CMakeFiles/xor-autobatch.dir/autobatch/train_xor-autobatch.cc.o
[ 71%] Building CXX object examples/CMakeFiles/read-write.dir/read-write/train_read-write.cc.o
[ 79%] Building CXX object examples/CMakeFiles/fflm.dir/fflm/train_fflm.cc.o
Scanning dependencies of target pydynet
[ 80%] Generating timestamp
running build_ext
INFO:root:Building Cython extensions...
INFO:root:INCLUDE_DIRS='/home/jisuk1/tools/dynet-base2/dynet /home/jisuk1/tools/dynet-base2/eigen'
INFO:root:LIBRARIES='dynet'
INFO:root:LIBRARY_DIRS='. /home/jisuk1/tools/dynet-base2/dynet/build/dynet/'
INFO:root:COMPILER_ARGS='-std=c++11 -Wno-unused-function'
INFO:root:EXTRA_LINK_ARGS="-Wl,-rpath='/usr/local/lib/',--no-as-needed"
INFO:root:RUNTIME_LIB_DIRS='/home/jisuk1/tools/dynet-base2/dynet/build/dynet/ /usr/local/lib/'
cythoning /home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.pyx to /home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp
[ 81%] Linking CXX executable 0_multiply
[ 81%] Built target 0_multiply
[ 82%] Linking CXX executable xor-batch
[ 83%] Linking CXX executable xor
/home/jisuk1/tools/dynet-base2/dynet/tutorial/1_linear_regression.cc: In function ‘int main(int, char**)’:
/home/jisuk1/tools/dynet-base2/dynet/tutorial/1_linear_regression.cc:21:35: warning: ‘normal.std::normal_distribution<float>::_M_saved’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     float y = 2 * x + normal(rng) * 0.33f;
                                   ^
[ 83%] Built target xor-batch
[ 83%] Built target xor
[ 84%] Linking CXX executable 1_linear_regression
[ 84%] Built target 1_linear_regression
[ 86%] Linking CXX executable xor-autobatch
[ 86%] Linking CXX executable read-write
[ 86%] Built target read-write
[ 86%] Built target xor-autobatch
[ 87%] Linking CXX executable fflm
[ 87%] Built target fflm
building '_dynet' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/home
creating build/temp.linux-x86_64-2.7/home/jisuk1
creating build/temp.linux-x86_64-2.7/home/jisuk1/tools
creating build/temp.linux-x86_64-2.7/home/jisuk1/tools/dynet-base2
creating build/temp.linux-x86_64-2.7/home/jisuk1/tools/dynet-base2/dynet
creating build/temp.linux-x86_64-2.7/home/jisuk1/tools/dynet-base2/dynet/build
creating build/temp.linux-x86_64-2.7/home/jisuk1/tools/dynet-base2/dynet/build/python
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/jisuk1/tools/dynet-base2/dynet -I/home/jisuk1/tools/dynet-base2/eigen -I/share/apps/include/python2.7 -c /home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp -o build/temp.linux-x86_64-2.7/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.o -std=c++11 -Wno-unused-function
[ 87%] Linking CXX executable embed-cl
[ 88%] Linking CXX executable rnnlm-aevb
[ 88%] Built target embed-cl
[ 89%] Linking CXX executable poisson-regression
[ 90%] Linking CXX executable rnnlm-cfsm
[ 90%] Built target rnnlm-aevb
[ 90%] Built target poisson-regression
[ 90%] Built target rnnlm-cfsm
[ 91%] Linking CXX executable mnist
[ 93%] Linking CXX executable tag-bilstm
[ 93%] Linking CXX executable imdb
[ 93%] Built target mnist
[ 94%] Linking CXX executable rnn-autobatch
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp: In function ‘PyObject* __pyx_f_6_dynet_16LookupParameters_init_from_array(__pyx_obj_6_dynet_LookupParameters*, PyObject*, int)’:
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp:16092:27: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   __pyx_t_7 = ((__pyx_t_6 > __pyx_v_self->thisptr.get_storage().values.size()) != 0);
                           ^
[ 96%] Built target tag-bilstm
[ 96%] Linking CXX executable rnnlm
[ 96%] Built target imdb
[ 98%] Linking CXX executable tok-embed
[ 99%] Linking CXX executable rnnlm-batch-nce
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp: In function ‘std::vector<unsigned int> __pyx_convert_vector_from_py_unsigned_int(PyObject*)’:
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp:77859:87: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         __pyx_t_5 = __Pyx_PyInt_As_unsigned_int(__pyx_v_item); if (unlikely(__pyx_t_5 == -1 && PyErr_Occurred())) __PYX_ERR(1, 52, __pyx_L1_error)
                                                                                       ^
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp:635:43: note: in definition of macro ‘unlikely’
   #define unlikely(x) __builtin_expect(!!(x), 0)
                                           ^
[ 99%] Built target rnn-autobatch
[ 99%] Linking CXX executable attention
[ 99%] Built target rnnlm-batch
[ 99%] Built target rnnlm
[ 99%] Built target tok-embed
[100%] Linking CXX executable encdec
[100%] Built target rnnlm-batch-nce
[100%] Built target attention
[100%] Built target encdec
g++ -pthread -shared build/temp.linux-x86_64-2.7/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.o -L. -L/home/jisuk1/tools/dynet-base2/dynet/build/dynet/ -L/share/apps/lib -Wl,-R/home/jisuk1/tools/dynet-base2/dynet/build/dynet/ -Wl,-R/usr/local/lib/ -ldynet -lpython2.7 -o /home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.so -Wl,-rpath='/usr/local/lib/',--no-as-needed
INFO:root:Copying built extensions...
[100%] Built target pydynet

Python installation log

jisuk1@dual-970-0-7:~/tools/dynet-base2/dynet/build/python$ python ../../setup.py build --build-dir=.. --skip-build install --user
running build
INFO:root:CMAKE_PATH='/share/apps/bin/cmake'
INFO:root:MAKE_PATH='/usr/bin/make'
INFO:root:MAKE_FLAGS='-j 4'
INFO:root:EIGEN3_INCLUDE_DIR='/home/jisuk1/tools/dynet-base2/eigen'
INFO:root:EIGEN3_DOWNLOAD_URL='https://bitbucket.org/eigen/eigen/get/699b6595fc47.zip';
INFO:root:CC_PATH='/share/apps/bin/gcc'
INFO:root:CXX_PATH='/share/apps/bin/g++'
INFO:root:SCRIPT_DIR='/home/jisuk1/tools/dynet-base2/dynet'
INFO:root:BUILD_DIR='/home/jisuk1/tools/dynet-base2/dynet/build'
INFO:root:INSTALL_PREFIX='/share/apps/lib/python2.7/site-packages/../../..'
INFO:root:PYTHON='/share/apps/bin/python'
cmake version 3.9.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
g++ (GCC) 5.4.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

running build_py
INFO:root:Building Python files...
creating build/lib.linux-x86_64-2.7
copying dynet.py -> build/lib.linux-x86_64-2.7
copying dynet_viz.py -> build/lib.linux-x86_64-2.7
copying dynet_config.py -> build/lib.linux-x86_64-2.7
running build_ext
INFO:root:Building Cython extensions...
INFO:root:INCLUDE_DIRS='/home/jisuk1/tools/dynet-base2/dynet /home/jisuk1/tools/dynet-base2/eigen /home/jisuk1/tools/dynet-base2/eigen'
INFO:root:LIBRARIES='dynet'
INFO:root:LIBRARY_DIRS='. /home/jisuk1/tools/dynet-base2/dynet/build//dynet/ /home/jisuk1/tools/dynet-base2/dynet/build/dynet/'
INFO:root:COMPILER_ARGS='-std=c++11 -Wno-unused-function'
INFO:root:EXTRA_LINK_ARGS="-Wl,-rpath='/usr/local/lib/',--no-as-needed"
INFO:root:RUNTIME_LIB_DIRS='/home/jisuk1/tools/dynet-base2/dynet/build//dynet/ /usr/local/lib/'
skipping '/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp' Cython extension (up-to-date)
building '_dynet' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/jisuk1/tools/dynet-base2/dynet -I/home/jisuk1/tools/dynet-base2/eigen -I/home/jisuk1/tools/dynet-base2/eigen -I/share/apps/include/python2.7 -c /home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp -o build/temp.linux-x86_64-2.7/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.o -std=c++11 -Wno-unused-function
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp: In function ‘PyObject* __pyx_f_6_dynet_16LookupParameters_init_from_array(__pyx_obj_6_dynet_LookupParameters*, PyObject*, int)’:
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp:16092:27: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   __pyx_t_7 = ((__pyx_t_6 > __pyx_v_self->thisptr.get_storage().values.size()) != 0);
                           ^
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp: In function ‘std::vector<unsigned int> __pyx_convert_vector_from_py_unsigned_int(PyObject*)’:
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp:77859:87: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         __pyx_t_5 = __Pyx_PyInt_As_unsigned_int(__pyx_v_item); if (unlikely(__pyx_t_5 == -1 && PyErr_Occurred())) __PYX_ERR(1, 52, __pyx_L1_error)
                                                                                       ^
/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.cpp:635:43: note: in definition of macro ‘unlikely’
   #define unlikely(x) __builtin_expect(!!(x), 0)
                                           ^
^[[A^[[A^[[A^[[Ag++ -pthread -shared build/temp.linux-x86_64-2.7/home/jisuk1/tools/dynet-base2/dynet/build/python/_dynet.o -L. -L/home/jisuk1/tools/dynet-base2/dynet/build//dynet/ -L/home/jisuk1/tools/dynet-base2/dynet/build/dynet/ -L/share/apps/lib -Wl,-R/home/jisuk1/tools/dynet-base2/dynet/build//dynet/ -Wl,-R/usr/local/lib/ -ldynet -lpython2.7 -o build/lib.linux-x86_64-2.7/_dynet.so -Wl,-rpath='/usr/local/lib/',--no-as-needed
INFO:root:Copying built extensions...
running install
running bdist_egg
running egg_info
creating dyNET.egg-info
writing requirements to dyNET.egg-info/requires.txt
writing dyNET.egg-info/PKG-INFO
writing top-level names to dyNET.egg-info/top_level.txt
writing dependency_links to dyNET.egg-info/dependency_links.txt
writing manifest file 'dyNET.egg-info/SOURCES.txt'
reading manifest file 'dyNET.egg-info/SOURCES.txt'
writing manifest file 'dyNET.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-2.7/dynet.py -> build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-2.7/_dynet.so -> build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-2.7/dynet_config.py -> build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-2.7/dynet_viz.py -> build/bdist.linux-x86_64/egg
byte-compiling build/bdist.linux-x86_64/egg/dynet.py to dynet.pyc
byte-compiling build/bdist.linux-x86_64/egg/dynet_config.py to dynet_config.pyc
byte-compiling build/bdist.linux-x86_64/egg/dynet_viz.py to dynet_viz.pyc
creating stub loader for _dynet.so
byte-compiling build/bdist.linux-x86_64/egg/_dynet.py to _dynet.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying dyNET.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dyNET.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dyNET.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dyNET.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dyNET.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/dyNET-0.0.0-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing dyNET-0.0.0-py2.7-linux-x86_64.egg
Removing /home/jisuk1/.local/lib/python2.7/site-packages/dyNET-0.0.0-py2.7-linux-x86_64.egg
Copying dyNET-0.0.0-py2.7-linux-x86_64.egg to /home/jisuk1/.local/lib/python2.7/site-packages
dyNET 0.0.0 is already the active version in easy-install.pth

Installed /home/jisuk1/.local/lib/python2.7/site-packages/dyNET-0.0.0-py2.7-linux-x86_64.egg
Processing dependencies for dyNET==0.0.0
Searching for numpy==1.13.1
Best match: numpy 1.13.1
Adding numpy 1.13.1 to easy-install.pth file

Using /share/apps/lib/python2.7/site-packages
Searching for Cython==0.25.1
Best match: Cython 0.25.1
Adding Cython 0.25.1 to easy-install.pth file
Installing cython script to /home/jisuk1/.local/bin
Installing cygdb script to /home/jisuk1/.local/bin
Installing cythonize script to /home/jisuk1/.local/bin

Using /share/apps/lib/python2.7/site-packages
Finished processing dependencies for dyNET==0.0.0

chunyang-wen commented 6 years ago

hmm. It seems that I have come across the same problem before. What I did was:

not adding any path to PATH or LD_LIBRARY_PATH (as you are under ubuntu)
add prefix/dynet/build/python/ to PYTHONPATH and export it.

Then try to

python -c 'import dynet;print dynet.__file__'

danielhers commented 6 years ago

@leejayyoon you are building /home/jisuk1/tools/dynet-base2/dynet but your LD_LIBRARY_PATH contains a different path, /home/jisuk1/tools/dynet-base/dynet, where you probably have an old version compiled.

leejayyoon commented 6 years ago

@danielhers Sorry for the confusion. It was simple error from copying paste from the open issue I made 20 days ago on #649, but I was trying this installation like 7 times and I changed the path accordingly. I am going to go ahead and modify original opening, as it might confuse others. Thanks for the comment.

But, what is the right way to clean all the dynet related materials to be sure? I just deleted under my ~/.local/lib and dynet git folder. (although I think 'rm -rf build' is fine)

leejayyoon commented 6 years ago

@chunyang-wen I did that, and it still spits out -->

  File "<string>", line 1, in <module>
  File "dynet.py", line 15, in <module>
    from _dynet import *
ImportError: ./_dynet.so: undefined symbol: _ZN5dynet13TextFileSaver4saveERKNS_19ParameterCollectionERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

When you see the directions on the python installation, it seems to indicate, at least in the dynet-base/dynet/build/python should be working regardless of the path or library issues, and that error I posted comes out from the python folder inside build folder.

I copy, pasted relevant part from http://dynet.readthedocs.io/en/latest/python.html. You now have a working Python binding inside of build/dynet. To verify this is working:

cd $PATH_TO_DYNET/build/python
python

then, within Python:

import dynet as dy
print dy.__version__
pc = dy.ParameterCollection()

chunyang-wen commented 6 years ago

cmake log:

-- The C compiler identification is GNU 4.8.4 -- The CXX compiler identification is GNU 4.8.4

Python installation log:

g++ (GCC) 5.4.0

You are using two different compilers?

neubig commented 6 years ago

@chunyang-wen Nice catch! This seems to be the problem.

leejayyoon commented 6 years ago

@chunyang-wen It finally works!! Thanks for catching that. I can't believe that I missed this as I was specifically looking for python gcc version mismatch ... but I guess I was only looking it on the python side. Thinking back, I should have checked the gcc on Cmake after recognizing that I cannot import dynet inside build/python folder... Thanks again.

Side note: Also it seems to be working without modification to PATH, LD_LIBRARY_PATH, DYLD_LIBRARY_PATH as @chunyang-wen has mentioned.

clab / dynet

Error on python installation #992