Error building TASO from source code

[cimeister]

I'm able to build the source code for TASO (albeit with the following warning that I'm not sure how to fix):

-- Configuring done
CMake Warning at CMakeLists.txt:69 (add_library):
  Cannot generate a safe linker search path for target taso_runtime because
  files in some directories may conflict with libraries in implicit
  directories:

    link library [libcublas.so] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /usr/lib/x86_64-linux-gnu/stubs

  Some of these libraries may not be found correctly.

-- Generating done
-- Build files have been written to: /tmp/taso/build

but once I try to install, I get this error:

...
[ 59%] Building CUDA object CMakeFiles/taso_runtime.dir/src/cudnn/concat_kernel.cu.o
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h:217:0:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /tmp/tmpxft_00003481_00000000-5_activation_kernel.cudafe1.stub.c:6:0,
                 from tmpxft_00003481_00000000-5_activation_kernel.cudafe1.stub.c:1:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h:217:0:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /tmp/tmpxft_0000348d_00000000-5_batchnorm_kernel.cudafe1.stub.c:6:0,
                 from tmpxft_0000348d_00000000-5_batchnorm_kernel.cudafe1.stub.c:1:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h:217:0:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /tmp/tmpxft_000034a3_00000000-5_cast_kernel.cudafe1.stub.c:6:0,
                 from tmpxft_000034a3_00000000-5_cast_kernel.cudafe1.stub.c:1:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
[ 61%] Building CUDA object CMakeFiles/taso_runtime.dir/src/cudnn/constant_kernel.cu.o
[ 62%] Building CUDA object CMakeFiles/taso_runtime.dir/src/cudnn/conv2d_kernel.cu.o
[ 64%] Building CUDA object CMakeFiles/taso_runtime.dir/src/cudnn/cuda_helper.cu.o
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h:217:0:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /tmp/tmpxft_000034c7_00000000-5_concat_kernel.cudafe1.stub.c:6:0,
                 from tmpxft_000034c7_00000000-5_concat_kernel.cudafe1.stub.c:1:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
/tmp/tmpxft_000034c7_00000000-5_concat_kernel.cudafe1.stub.c: In function ‘void __device_stub__Z18assign_with_stridePfPKfiii(_ZN4taso8DATATYPEE*, const _ZN4taso8DATATYPEE*, int, int, int)’:
/tmp/tmpxft_000034c7_00000000-5_concat_kernel.cudafe1.stub.c:12:149: error: ‘__args_arr’ was not declared in this scope
 void __device_stub__Z18assign_with_stridePfPKfiii(_ZN4taso8DATATYPEE *__par0, const _ZN4taso8DATATYPEE *__par1, int __par2, int __par3, int __par4){__cudaSetupArgSimple(__par0, 0UL);__cudaSetupArgSimple(__par1, 8UL);__cudaSetupArgSimple(__par2, 16UL);__cudaSetupArgSimple(__par3, 20UL);__cudaSetupArgSimple(__par4, 24UL);__cudaLaunch(((char *)((void ( *)(_ZN4taso8DATATYPEE *, const _ZN4taso8DATATYPEE *, int, int, int))assign_with_stride)));}
                                                                                                                                                     ^
/tmp/tmpxft_000034c7_00000000-5_concat_kernel.cudafe1.stub.c:12:149: error: ‘__args_idx’ was not declared in this scope
 void __device_stub__Z18assign_with_stridePfPKfiii(_ZN4taso8DATATYPEE *__par0, const _ZN4taso8DATATYPEE *__par1, int __par2, int __par3, int __par4){__cudaSetupArgSimple(__par0, 0UL);__cudaSetupArgSimple(__par1, 8UL);__cudaSetupArgSimple(__par2, 16UL);__cudaSetupArgSimple(__par3, 20UL);__cudaSetupArgSimple(__par4, 24UL);__cudaLaunch(((char *)((void ( *)(_ZN4taso8DATATYPEE *, const _ZN4taso8DATATYPEE *, int, int, int))assign_with_stride)));}
                                                                                                                                                     ^
CMakeFiles/taso_runtime.dir/build.make:862: recipe for target 'CMakeFiles/taso_runtime.dir/src/cudnn/concat_kernel.cu.o' failed
make[2]: *** [CMakeFiles/taso_runtime.dir/src/cudnn/concat_kernel.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h:217:0:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /tmp/tmpxft_000034ed_00000000-5_constant_kernel.cudafe1.stub.c:6:0,
                 from tmpxft_000034ed_00000000-5_constant_kernel.cudafe1.stub.c:1:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h:217:0:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /tmp/tmpxft_00003517_00000000-5_cuda_helper.cudafe1.stub.c:6:0,
                 from tmpxft_00003517_00000000-5_cuda_helper.cudafe1.stub.c:1:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h:217:0:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from /tmp/tmpxft_00003503_00000000-5_conv2d_kernel.cudafe1.stub.c:6:0,
                 from tmpxft_00003503_00000000-5_conv2d_kernel.cudafe1.stub.c:1:
/usr/local/cuda-10.0/include/crt/host_runtime.h:19:2: warning: #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "crt/device_functions.h is an internal header file and must not be used directly.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
/tmp/tmpxft_00003517_00000000-5_cuda_helper.cudafe1.stub.c: In function ‘void __device_stub__Z13assign_kernelPfif(float*, int, float)’:
/tmp/tmpxft_00003517_00000000-5_cuda_helper.cudafe1.stub.c:12:83: error: ‘__args_arr’ was not declared in this scope
 void __device_stub__Z13assign_kernelPfif(float *__par0, int __par1, float __par2){__cudaSetupArgSimple(__par0, 0UL);__cudaSetupArgSimple(__par1, 8UL);__cudaSetupArgSimple(__par2, 12UL);__cudaLaunch(((char *)((void ( *)(float *, int, float))assign_kernel)));}
                                                                                   ^
/tmp/tmpxft_00003517_00000000-5_cuda_helper.cudafe1.stub.c:12:83: error: ‘__args_idx’ was not declared in this scope
 void __device_stub__Z13assign_kernelPfif(float *__par0, int __par1, float __par2){__cudaSetupArgSimple(__par0, 0UL);__cudaSetupArgSimple(__par1, 8UL);__cudaSetupArgSimple(__par2, 12UL);__cudaLaunch(((char *)((void ( *)(float *, int, float))assign_kernel)));}
                                                                                   ^
/tmp/tmpxft_00003517_00000000-5_cuda_helper.cudafe1.stub.c: In function ‘void __device_stub__Z11copy_kernelPfPKfi(float*, const float*, int)’:
/tmp/tmpxft_00003517_00000000-5_cuda_helper.cudafe1.stub.c:1:96: error: ‘__args_arr’ was not declared in this scope
 #pragma GCC diagnostic push
                                                                                                ^
/tmp/tmpxft_00003517_00000000-5_cuda_helper.cudafe1.stub.c:1:96: error: ‘__args_idx’ was not declared in this scope
 #pragma GCC diagnostic push
                                                                                                ^
CMakeFiles/taso_runtime.dir/build.make:934: recipe for target 'CMakeFiles/taso_runtime.dir/src/cudnn/cuda_helper.cu.o' failed
make[2]: *** [CMakeFiles/taso_runtime.dir/src/cudnn/cuda_helper.cu.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/taso_runtime.dir/all' failed
make[1]: *** [CMakeFiles/taso_runtime.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

the problems seem unrelated but I thought I'd include the warnings to provide all the context.

Thanks for your help!

[jiazhihao]

It seems there are potential misconfiguration when you did cmake. And people have experienced similar issues with onnx https://github.com/onnx/onnx-tensorrt/issues/150. Can you try to set USE_CUDA to your CUDA path in file config.cmake, and redo cmake.

[cimeister]

The same error still occurs, unfortunately