Open loretoparisi opened 7 years ago
[UPDATE]
I have found a solution for eigen3
:
RUN curl http://security.ubuntu.com/ubuntu/pool/universe/e/eigen3/libeigen3-dev_3.3.2-1_all.deb -o libeigen3-dev_3.3.2-1_all.deb && dpkg -i libeigen3-dev_3.3.2-1_all.deb
so using dpkg
solves the install dir issue:
-- Could NOT find Boost
-- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.3")
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "6.5")
-- Could NOT find MKL (missing: MKL_INCLUDE_DIR MKL_INTEL_LP64_LIBRARY MKL_GF_LP64_LIBRARY MKL_GNU_THREAD_LIBRARY MKL_CORE_LIBRARY)
The problem with boost
is still present, so of course it still exits with error later on.
Reading the apt-get
logs I can see that it installs some packages of boost
but not all the recommended
ones:
Reading state information...
The following additional packages will be installed:
libboost1.58-dev
Suggested packages:
libboost-doc libboost1.58-doc libboost-atomic1.58-dev
libboost-chrono1.58-dev libboost-context1.58-dev libboost-coroutine1.58-dev
libboost-date-time1.58-dev libboost-exception1.58-dev
libboost-filesystem1.58-dev libboost-graph1.58-dev
libboost-graph-parallel1.58-dev libboost-iostreams1.58-dev
libboost-locale1.58-dev libboost-log1.58-dev libboost-math1.58-dev
libboost-mpi1.58-dev libboost-mpi-python1.58-dev
libboost-program-options1.58-dev libboost-python1.58-dev
libboost-random1.58-dev libboost-regex1.58-dev
libboost-serialization1.58-dev libboost-signals1.58-dev
libboost-system1.58-dev libboost-test1.58-dev libboost-thread1.58-dev
libboost-timer1.58-dev libboost-wave1.58-dev libboost1.58-tools-dev
libmpfrc++-dev libntl-dev
The following NEW packages will be installed:
libboost-dev libboost1.58-dev pkg-config
0 upgraded, 3 newly installed, 0 to remove and 9 not upgraded.
Not sure if this can be the issue here.
[UPDATE]
Ok this is my last finding to solve the boost
issue:
# OpenNMT dependencies: boot, eigen, intel mkl
RUN apt-get update && apt-get install -y \
libboost-dev \
libboost-all-dev \
pkg-config
It seems that libboost-all-dev
will install all needed dependencies not only the boost
library alone, so that now I came out with
-- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.3")
...
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "6.5")
-- Could NOT find MKL (missing: MKL_INCLUDE_DIR MKL_INTEL_LP64_LIBRARY MKL_GF_LP64_LIBRARY MKL_GNU_THREAD_LIBRARY MKL_CORE_LIBRARY)
-- Boost version: 1.58.0
I just have some complains about the architecture from nvcc
when building
[ 27%] Building C object lib/TH/CMakeFiles/TH.dir/THGeneral.c.o
[ 30%] Building C object lib/TH/CMakeFiles/TH.dir/THFile.c.o
[ 33%] Building C object lib/TH/CMakeFiles/TH.dir/THDiskFile.c.o
[ 36%] Linking C shared library libTH.so
[ 36%] Built target TH
[ 39%] Building NVCC (Device) object CMakeFiles/onmt.dir/src/cuda/onmt_generated_Kernels.cu.o
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
and the missing Intel MKL, but this is a separate thing.
If you want to make the nvcc
warnings quiet, you can pass this flag to cmake
:
-DCUDA_NVCC_FLAGS="-gencode=arch=compute_61,code=compute_61"
where 61
is the compute capability of your GPU.
Intel MKL is optional.
@guillaumekln thanks it make sense, by example when I have built the Dockerfile for darkflow
I add to change this manually in the Cmake.
Somewhere I have used a command to detected the architecture automatically, I'm searching this right now...
Closing this since it has been solved btw.
@guillaumekln re-opening since I get
root@18b64b3f8bc0:/mxmlib/CTranslate/build/cli# echo "hello this is me" | ./translate --model /root/nmt/onmt_baseline_wmt15-all.en-de_epoch13_7.19_release.t7 --beam_size 5 --batch_size 1 -
Illegal instruction (core dumped)
where the built file was
root@18b64b3f8bc0:/mxmlib/CTranslate/build/cli# file translate
translate: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9116d7fc8728d04bb935c6781256aa9c486e6358, not stripped
My cmake
options were
cmake -DCMAKE_CXX_FLAGS="-march=native" -DEIGEN_ROOT="/usr/include/eigen3" .. &&
I have added the DWITH_OPENMP
and it seems to work
cmake -DWITH_OPENMP=OFF -DCMAKE_CXX_FLAGS="-march=native" -DEIGEN_ROOT="/usr/include/eigen3" .. &&
What is your g++ --version
?
@guillaumekln
0.0.90.190 ubuntu@ai-tools-nmt-i-04fc6623dcd3531f8:~$ g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.4' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)
Not sure about this.
g++
?@guillaumekln I think the problem were the linked libraries
root@ac0117818001:/mxmlib/CTranslate/build/cli# ldd translate
linux-vdso.so.1 => (0x00007fff2bbda000)
libonmt.so => /mxmlib/CTranslate/build/libonmt.so (0x00007f7cabc49000)
libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007f7cab9b9000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7cab637000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7cab421000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7cab057000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7caae3a000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7caac36000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7caaa2d000)
libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007f7ca7a12000)
libOpenNMTTokenizer.so => /mxmlib/CTranslate/build/lib/tokenizer/libOpenNMTTokenizer.so (0x00007f7ca77f9000)
libTH.so => /mxmlib/CTranslate/build/lib/TH/libTH.so (0x00007f7ca75f0000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7ca72e7000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f7ca70c5000)
/lib64/ld-linux-x86-64.so.2 (0x0000557438c78000)
So I think I was missing
libTH.so => /mxmlib/CTranslate/build/lib/TH/libTH.so (0x00007f7ca75f0000)
libOpenNMTTokenizer.so => /mxmlib/CTranslate/build/lib/tokenizer/libOpenNMTTokenizer.so (0x00007f7ca77f9000)
libonmt.so => /mxmlib/CTranslate/build/libonmt.so (0x00007f7cabc49000)
or libboost
compiled on a different machine.
Question: no way to get rid of boost
and eigen
i.e. not using them?
Why would you like to drop these dependencies?
@guillaumekln I wonder if it would be possibile to make the binaries standalone i.e. like fasttext
does. This would be great and make everything more portable. Thank you!
Did you try to compile with static libraries?
cmake -DBUILD_SHARED_LIBS=OFF [...]
@guillaumekln thanks now I have only the boost
dependency:
root@e195516272c0:/mxmlib/CTranslate/build/cli# ldd translate
linux-vdso.so.1 => (0x00007ffc8f1cc000)
libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007ffa56042000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffa55e24000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffa55c20000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffa55a18000)
libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007ffa529fc000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffa5267a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffa52371000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffa5215a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffa51d91000)
/lib64/ld-linux-x86-64.so.2 (0x00005637afd00000)
boost
is only used for managing command line options. It should be easy to write your own translation client without this dependency.
See cli/translate.cc
.
I have successfully installed both Boost and Eigen on
Ubuntu 16.04 LTS
:so
libeigen3-dev (3.3~beta1-2)
andlibboost1.58-dev:amd64 (1.58.0+dfsg-5ubuntu3.1)
.but when I compile
the
cmake
does not detect boost or eigen3 in the system default folder.Since I'm building on
docker
there are not previous packages installed, so it should find the ones I have installed viaapt-get
:where
-t xenial
is necessary to install the latest3.3-beta
.