CMake Error: could not generate ./dynet_generated_gpu-merge2.cu.o

[zerovirus123]

I followed the instructions outlined in Dynet docs. These are my setups.

CMake Command cmake .. -DEIGEN3_INCLUDE_DIR=/usr/local/include/eigen -DENABLE_CPP_EXAMPLES=ON -DBACKEND=cuda -DCUDNN_ROOT=/usr/local/cuda/lib64 -DENABLE_BOOST=ON -DPYTHON=which python

Dependencies
g++ version
g++ (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0

CUDA Version
/usr/local/cuda-10.1/

CUDNN Version
7.6.5

Boost Version
Version: 1.65.1.0ubuntu1

Python Version
/home/briansia/anaconda3/bin/python
Python 3.7.3

Eigen Version
3.3.90

Running the CMake produces the following output.

-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 8.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Optimization level: fast
-- Enabling Boost
-- Found Boost: /usr/include (found version "1.65.1") found components: program_options regex serialization 
-- Boost dir is /usr/include
-- BACKEND: cuda
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /usr/local/cuda (found version "10.1") 
CUDA_LIBRARIES: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so;/usr/local/cuda/lib64/libcurand.so
-- Found CUDNN (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Successfully include CUDNN flags
-- Eigen dir is /usr/local/include/eigen
-- Found Cython version 0.29.15

--- CUDA: CUBLAS: /usr/lib/x86_64-linux-gnu/libcublas.so RT: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so;/usr/local/cuda/lib64/libcurand.so;/usr/local/cuda/lib64/libcudnn.so
CMAKE_INSTALL_PREFIX="/usr/local"
PROJECT_SOURCE_DIR="/home/briansia/dynet"
PROJECT_BINARY_DIR="/home/briansia/dynet/build"
LIBS="/usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.65.1\;/usr/lib/x86_64-linux-gnu/libboost_regex.so.1.65.1\;/usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.65.1\;/usr/local/cuda/lib64/libcudart_static.a\;Threads::Threads\;dl\;/usr/lib/x86_64-linux-gnu/librt.so\;/usr/local/cuda/lib64/libcurand.so\;-lpthread"
EIGEN3_INCLUDE_DIR="/usr/local/include/eigen"
MKL_LINK_DIRS=""
WITH_CUDA_BACKEND="1"
CUDA_RT_FILES="libcudart_static.a\;Threads::Threads\;dl\;librt.so\;libcurand.so\;libcudnn.so"
CUDA_RT_DIRS="/usr/local/cuda/lib64\;/usr/lib/x86_64-linux-gnu"
CUDA_CUBLAS_FILES="libcublas.so"
CUDA_CUBLAS_DIRS="/usr/lib/x86_64-linux-gnu"
MSVC=""
-- Found Boost: /usr/include (found version "1.65.1") found components: system filesystem unit_test_framework 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/briansia/dynet/build

I could not find out why CMAKE_HAVE_LIBC_PTHREAD failed and pthread_create could not be found in the CUDA backend. However, I proceeded to run make anyways.

Here is the output from make -j 12.

make.log

The very end of the log showed that make failed because of the following lines.

dynet/CMakeFiles/dynet.dir/build.make:97: recipe for target 'dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge0.cu.o' failed

[lots of ptxas error thrown here]

Error generating file
  /home/briansia/dynet/build/dynet/CMakeFiles/dynet.dir//./dynet_generated_gpu-merge2.cu.o

Does anyone have an idea as to what caused this? Thanks.

[cydur]

In Issue https://github.com/clab/dynet/issues/1598 they proposed to use at least CMake 3.12.2. Did you try this?

[zerovirus123]

I used CMake 3.17 installed using snap. Do you think that could be the issue?

[cydur]

Sorry I have no clue. I tried myself today with cmake 3.17.3 CUDA 10.2 CUDNN 8.0.1.13 and get the following errors: /home/cyril/projects/dynet/dynet/cudnn-ops.cu(107): error: identifier "CUDNN_CONVOLUTION_FWD_PREFER_FASTEST" is undefined

/home/cyril/projects/dynet/dynet/cudnn-ops.cu(107): error: identifier "cudnnGetConvolutionForwardAlgorithm" is undefined

/home/cyril/projects/dynet/dynet/cudnn-ops.cu(190): error: identifier "CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST" is undefined

/home/cyril/projects/dynet/dynet/cudnn-ops.cu(190): error: identifier "cudnnGetConvolutionBackwardDataAlgorithm" is undefined

/home/cyril/projects/dynet/dynet/cudnn-ops.cu(209): error: identifier "CUDNN_CONVOLUTION_BWD_FILTER_PREFER_FASTEST" is undefined

/home/cyril/projects/dynet/dynet/cudnn-ops.cu(209): error: identifier "cudnnGetConvolutionBackwardFilterAlgorithm" is undefined

6 errors detected in the compilation of "/tmp/tmpxft_00004787_00000000-6_cudnn-ops.cpp1.ii". CMake Error at dynet_generated_cudnn-ops.cu.o.Release.cmake:276 (message): Error generating file /home/cyril/projects/dynet/build_cudnn_release/dynet/CMakeFiles/dynet.dir//./dynet_generated_cudnn-ops.cu.o

dynet/CMakeFiles/dynet.dir/build.make:625: recipe for target 'dynet/CMakeFiles/dynet.dir/dynet_generated_cudnn-ops.cu.o' failed

Does this look similar to yours?

[cydur]

I fixed my above errors. I don't know if this also fixes your errors. But with the following versions and compile flags I can compile dynet and it runs well: System: Ubuntu 18.04 LTS Compiler: gcc 8.4.0 cmake 3.17.3 CUDA 10.2.89-440.33.01_1.0-1 CUDNN 8.0.1.13 Eigen: 3.3.90 MKL: 2020.0.166

Cmake options (maybe you have different installation paths and GPU aschitecture): -DCMAKE_BUILD_TYPE=Release -DRELEASE_OPT_LEVEL=3 -DCMAKE_CXX_FLAGS="-Wno-invalid-partial-specialization -Wno-c++11-narrowing " -DEIGEN3_INCLUDE_DIR=/usr/local/include/eigen3 -DMKL=TRUE -DMKL_ROOT=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl -DENABLE_BOOST=ON -DBACKEND=cuda -DCUDA_ARCH=75

With the above Eigen version you will get millions of warnings. But nevertheless it works well.