search

deephealthproject / pyeddl

Python wrapper for the EDDL library.
MIT License
13 stars 2 forks source link

Errors in build scripts using CMake to install eddl after patch 0.4 #20

Closed Andrea-Oliveri closed 4 years ago

Andrea-Oliveri commented 4 years ago

Good Morning, since the release of patch 0.4 the scripts I previously used to build and install eddl and pyeddl do not work anymore. To create the scripts I simply looked at the guide for installing eddl with CMake on the pyeddl github. I previously compiled eddl with CMake with these parameters -DEDDL_SHARED=ON -DBUILD_TESTS=ON -DBUILD_TARGET=CPU -DBUILD_EXAMPLES=ON -DCMAKE_INSTALL_PREFIX=/home/$USER/eddl.

I tried to modify my scripts in several ways: I tried to create a conda environment with the .yml in the eddl repository (success) then compiled and installed eddl with CMake (success), then went into the pyeddl directory and asked conda to install the pyeddl dependencies: pytest, pybind (success if no other package is installed (failure to solve environment is any of pandas, matplotlib, keras, tensorflow, scikit-learn, scipy also have to be installed)). When running python3 setup.py install if fails with:

running bdist_egg
running egg_info
writing pyeddl.egg-info/PKG-INFO
writing dependency_links to pyeddl.egg-info/dependency_links.txt
writing requirements to pyeddl.egg-info/requires.txt
writing top-level names to pyeddl.egg-info/top_level.txt
reading manifest file 'pyeddl.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyeddl.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
building 'pyeddl._core' extension
gcc -pthread -B /home/andrea/anaconda3/envs/eddl/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Isrc -I/home/andrea/anaconda3/envs/eddl/include -I/home/andrea/anaconda3/envs/eddl/include -I/home/andrea/eddl/include -I/home/andrea/anaconda3/envs/eddl/include/python3.8 -c src/_core.cpp -o build/temp.linux-x86_64-3.8/src/_core.o -std=c++11 -fvisibility=hidden -DEDDL_WITH_PROTOBUF
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from src/_core.cpp:2:0:
/home/andrea/eddl/include/eddl/descriptors/tensor_descriptors.h:18:10: fatal error: Eigen/Dense: No such file or directory
 #include <Eigen/Dense>
          ^~~~~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1

Then I tried to do everything in two conda commands and install pyeddl:

conda create -y -n eddl tensorflow=1.14 tensorflow-gpu=1.14 keras=2.2.4 keras-gpu=2.2.4 matplotlib numpy pandas scikit-learn scipy pybind11 pytest

conda install -n eddl -c salvacarrion eddl

cd pyeddl
python3 setup.py install

which successfully installs the whole conda environment but fails at the installation of pyeddl with the same error as earlier.

A reason why it would be interesting to keep compiling eddl with CMake is the possibility to do so for GPU, as I do not know by installing with conda what variant it installs but my guess would be CPU. Please let me know if you plan to release a conda package eddl-gpu and one for pyeddl. As to know, I don't see a way I can make the library successfully. Please let me know if there is anything I can do.

Thank you in advance.

simleo commented 4 years ago

Hi,

to build the EDDL shared library now you have to set -DBUILD_SHARED_LIB=ON, not -DEDDL_SHARED=ON. I previously forgot to update this information in README.md, but I just pushed an update.

The README.md has also been updated to include more info on Eigen. Recent (>=0.4) versions of EDDL do not include Eigen as a git submodule anymore, so it must be installed as a dependency. For instance, apt-get install libeigen3-dev. Also note that EDDL code includes Eigen headers like in this example: #include <Eigen/Dense>, e.g., with Eigen as the root directory. However, Eigen installations usually have the header rooted at eigen3 (for instance, the apt installation places them in /usr/include/eigen3). To work around this you can either add a symlink or set CPATH, e.g., export CPATH="/usr/include/eigen3:${CPATH}". Finally, the current version of Eigen installed by apt has issues with CUDA. If you are compiling for GPU, install a recent version of Eigen from source. If you install from source you will probably get the headers installed into /usr/local/include/eigen3, so the above env setting becomes export CPATH="/usr/local/include/eigen3:${CPATH}".

The Docker setup used for pyeddl testing can also be used as an installation guide. Take a look at:

Andrea-Oliveri commented 4 years ago

Thank you, I managed to install pyeddl and eddl both by compiling eddl and by installing it through conda. However, when running pytest tests form the pytest folder all tests fail with:

========================================== test session starts ===========================================
platform linux -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /home/andrea/Desktop
collected 0 items / 4 errors                                                                             

================================================= ERRORS =================================================
__________________________________ ERROR collecting tests/test_eddl.py ___________________________________
ImportError while importing test module '/home/andrea/Desktop/tests/test_eddl.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
Projet_de_Batchelor/repo/pyeddl/tests/test_eddl.py:21: in <module>
    import pyeddl._core.eddl as eddl
E   ImportError: /home/andrea/anaconda3/envs/eddl/lib/python3.7/site-packages/pyeddl-0.4.0-py3.7-linux-x86_64.egg/pyeddl/_core.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN4eddl21save_net_to_onnx_fileEP3NetNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
__________________________________ ERROR collecting tests/test_eddlT.py __________________________________
ImportError while importing test module '/home/andrea/Desktop/tests/test_eddlT.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
Projet_de_Batchelor/repo/pyeddl/tests/test_eddlT.py:22: in <module>
    import pyeddl._core.eddlT as eddlT
E   ImportError: /home/andrea/anaconda3/envs/eddl/lib/python3.7/site-packages/pyeddl-0.4.0-py3.7-linux-x86_64.egg/pyeddl/_core.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN4eddl21save_net_to_onnx_fileEP3NetNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
_____________________________ ERROR collecting tests/test_py_loss_metric.py ______________________________
ImportError while importing test module '/home/andrea/Desktop/tests/test_py_loss_metric.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
Projet_de_Batchelor/repo/pyeddl/tests/test_py_loss_metric.py:23: in <module>
    import pyeddl._core.eddl as eddl
E   ImportError: /home/andrea/anaconda3/envs/eddl/lib/python3.7/site-packages/pyeddl-0.4.0-py3.7-linux-x86_64.egg/pyeddl/_core.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN4eddl21save_net_to_onnx_fileEP3NetNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
_________________________________ ERROR collecting tests/test_tensor.py __________________________________
ImportError while importing test module '/home/andrea/Desktop/tests/test_tensor.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
Projet_de_Batchelor/repo/pyeddl/tests/test_tensor.py:22: in <module>
    from pyeddl._core import Tensor
E   ImportError: /home/andrea/anaconda3/envs/eddl/lib/python3.7/site-packages/pyeddl-0.4.0-py3.7-linux-x86_64.egg/pyeddl/_core.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN4eddl21save_net_to_onnx_fileEP3NetNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================== 4 errors in 0.40s ============================================

Once more, thank you in advance.

simleo commented 4 years ago

Judging from the output above, I guess you are getting these errors with the conda eddl installation. I haven't tried that, but since the import is failing due to a missing onnx-related symbol, my guess is that the eddl library you have installed has been compiled without onnx support. When building eddl with cmake, you have to explicitly enable onnx support by setting -DBUILD_PROTOBUF=ON. I'm not sure about the conda eddl releases (you can ask the eddl guys about them), but from what I see there are currently three: eddl, eddl-gpu and eddl-gpu-onnx, and I think only the latter has onnx support.

On the other hand, if you do not need onnx functionality in pyeddl, you can turn it off by setting the EDDL_WITH_PROTOBUF environment variable to OFF or FALSE before building pyeddl. If you do this, then pyeddl will work even with an eddl that's been compiled without onnx support.

Andrea-Oliveri commented 4 years ago

Thank you very much, everything seems to work now despite I had to deactivate Protobuf.