[tsdf_node-3] process has died [pid 8271, exit code -11

WuRobotics commented 3 years ago

Info of my PC: Operating system version: Ubuntu 16.04.7 ROS version: ROS Kinetic Graphics card type: RTX 3070 (Desktop version) (Core i7 10700)(16GB ram) Nvidia driver version: 470.63.01 CUDA version: V11.1.74 cmake version: Cmake 3.11

Issue description: Hi, I'm trying to run the Yak_ros demo on my newly purchased PC with RTX 3070 GPU. According to the ReadMe file of "Yak" and "Yak_ros", the workspace was successfully compiled with some warnings. I source the "setup.bash" file in the "devel" folder and tried to run the demo, I got the error of : [tsdf_node-3] process has died

roslaunch yak_ros demo.launch: Selection_001

Nvidia driver version: Selection_002

Cuda version: Selection_003

Catkin build output Selection_004

Selection_005

Selection_006

My attempts to solve the problem Attempt 1:
As shown in the last image, there are some warnings related to CUDA, I'm wondering maybe the CUDA 11.1 is not compatible with the Yak, or some functions in CUDA 11.1 are different from the lower version CUDA. So I tried to downgrade the CUDA version to CUDA 9.0. After installation the CUDA 9.0

I found that the RTX 3070 is not compatible with CUDA 9.0, the Yak workspace can not be generated with Catkin build I tried to solve the problem with: export TORCH_CUDA_ARCH_LIST="8.0" According to https://github.com/open-mmlab/OpenPCDet/issues/591, but still no luck.

Attempt 2:
I tried to downgrade the GPU driver version and CUDA together, but still no luck.

Other info The PCL version is 1.9.1, compiled from source. The Cmake version is 3.11, compiled from source. Other dependencies related withYak and Yak_ros ( the gl_depth_sim package, ros_industrial_cmake_boilerplate ) are working normally.

Could you please help me to check the problem and give me a hint? Many thanks for your help!

@schornakj @gavanderhoorn @gachiemchiep

WuRobotics commented 3 years ago

The Catkin build output is shown below in script format:

`robotking@robotking-XPS-8940:~/yak_ros_test$ catkin build

Profile: default Extending: [env] /opt/ros/kinetic Workspace: /home/robotking/yak_ros_test

Build Space: [exists] /home/robotking/yak_ros_test/build Devel Space: [exists] /home/robotking/yak_ros_test/devel Install Space: [unused] /home/robotking/yak_ros_test/install Log Space: [missing] /home/robotking/yak_ros_test/logs Source Space: [exists] /home/robotking/yak_ros_test/src DESTDIR: [unused] None

Devel Space Layout: linked Install Space Layout: None

Additional CMake Args: None Additional Make Args: None Additional catkin Make Args: None Internal Make Job Server: True Cache Job Environments: False

Whitelisted Packages: None Blacklisted Packages: None

Workspace configuration appears valid.

[build] Found '4' packages in 0.0 seconds.
[build] Updating package table.
Warning: generated devel space setup files have been deleted. Starting >>> catkin_tools_prebuild
Finished <<< catkin_tools_prebuild [ 1.4 seconds ]
Starting >>> gl_depth_sim
Starting >>> ros_industrial_cmake_boilerplate
Finished <<< ros_industrial_cmake_boilerplate [ 0.6 seconds ]
Starting >>> yak
Starting >>> yak_ext
Finished <<< yak_ext [ 0.6 seconds ]

Warnings << yak:make /home/robotking/yak_ros_test/logs/yak/build.make.000.log

:0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name :0:20: warning: missing whitespace after the macro name /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In destructor ‘kfusion::cuda::TextureBinder::~TextureBinder()’: /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:49:75: warning: ‘cudaError_t cudaUnbindTexture(const textureReference*)’ is deprecated [-Wdeprecated-declarations] /usr/local/cuda-11.1/include/cuda_runtime_api.h:8060:46: note: declared here extern __CUDA_DEPRECATED __host__ cudaError_t CUDARTAPI cudaUnbindTexture(const struct textureReference *texref); ^ /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In instantiation of ‘kfusion::cuda::TextureBinder::TextureBinder(const kfusion::cuda::DeviceArray2D&, const texture&) [with T = short unsigned int; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’: /home/robotking/yak_ros_test/src/yak/yak/src/cuda/proj_icp.cu:395:50: required from here /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:18:49: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const texture&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = short unsigned int; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’ is deprecated [-Wdeprecated-declarations] cudaSafeCall(cudaBindTexture2D(0, tex, arr.ptr(), desc, arr.cols(), arr.rows(), arr.step())); ^ /usr/local/cuda-11.1/include/cuda_runtime.h:1396:53: note: declared here static __CUDA_DEPRECATED __inline__ __host__ cudaError_t cudaBindTexture2D( ^ /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In instantiation of ‘kfusion::cuda::TextureBinder::TextureBinder(const kfusion::cuda::DeviceArray2D&, const texture&) [with T = float4; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’: /home/robotking/yak_ros_test/src/yak/yak/src/cuda/proj_icp.cu:396:50: required from here /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:18:49: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const texture&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = float4; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’ is deprecated [-Wdeprecated-declarations] cudaSafeCall(cudaBindTexture2D(0, tex, arr.ptr(), desc, arr.cols(), arr.rows(), arr.step())); ^ /usr/local/cuda-11.1/include/cuda_runtime.h:1396:53: note: declared here static __CUDA_DEPRECATED __inline__ __host__ cudaError_t cudaBindTexture2D( ^ /usr/local/cuda-11.1/include/cuda_runtime.h: In instantiation of ‘cudaError_t cudaBindTexture2D(size_t*, const texture&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = short unsigned int; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’: /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:18:49: required from ‘kfusion::cuda::TextureBinder::TextureBinder(const kfusion::cuda::DeviceArray2D&, const texture&) [with T = short unsigned int; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’ /home/robotking/yak_ros_test/src/yak/yak/src/cuda/proj_icp.cu:395:50: required from here /usr/local/cuda-11.1/include/cuda_runtime.h:1406:27: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const textureReference*, const void*, const cudaChannelFormatDesc*, size_t, size_t, size_t)’ is deprecated [-Wdeprecated-declarations] return ::cudaBindTexture2D(offset, &tex, devPtr, &desc, width, height, pitch); ^ /usr/local/cuda-11.1/include/cuda_runtime_api.h:7956:46: note: declared here extern __CUDA_DEPRECATED __host__ cudaError_t CUDARTAPI cudaBindTexture2D(size_t *offset, const struct textureReference *texref, const void *devPtr, const struct cudaChannelFormatDesc *desc, size_t width, size_t height, size_t pitch); ^ /usr/local/cuda-11.1/include/cuda_runtime.h: In instantiation of ‘cudaError_t cudaBindTexture2D(size_t*, const texture&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = float4; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’: /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:18:49: required from ‘kfusion::cuda::TextureBinder::TextureBinder(const kfusion::cuda::DeviceArray2D&, const texture&) [with T = float4; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’ /home/robotking/yak_ros_test/src/yak/yak/src/cuda/proj_icp.cu:396:50: required from here /usr/local/cuda-11.1/include/cuda_runtime.h:1406:27: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const textureReference*, const void*, const cudaChannelFormatDesc*, size_t, size_t, size_t)’ is deprecated [-Wdeprecated-declarations] return ::cudaBindTexture2D(offset, &tex, devPtr, &desc, width, height, pitch); ^ /usr/local/cuda-11.1/include/cuda_runtime_api.h:7956:46: note: declared here extern __CUDA_DEPRECATED __host__ cudaError_t CUDARTAPI cudaBindTexture2D(size_t *offset, const struct textureReference *texref, const void *devPtr, const struct cudaChannelFormatDesc *desc, size_t width, size_t height, size_t pitch); ^ /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In destructor ‘kfusion::cuda::TextureBinder::~TextureBinder()’: /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:49:75: warning: ‘cudaError_t cudaUnbindTexture(const textureReference*)’ is deprecated [-Wdeprecated-declarations] /usr/local/cuda-11.1/include/cuda_runtime_api.h:8060:46: note: declared here extern __CUDA_DEPRECATED __host__ cudaError_t CUDARTAPI cudaUnbindTexture(const struct textureReference *texref); ^ /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In instantiation of ‘kfusion::cuda::TextureBinder::TextureBinder(const A&, const texture&, const cudaChannelFormatDesc&) [with A = kfusion::cuda::PtrStepSz; T = float; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’: /home/robotking/yak_ros_test/src/yak/yak/src/cuda/tsdf_volume.cu:131:73: required from here /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:39:49: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const texture&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = float; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’ is deprecated [-Wdeprecated-declarations] cudaSafeCall(cudaBindTexture2D(0, tex, arr.data, desc, arr.cols, arr.rows, arr.step)); ^ /usr/local/cuda-11.1/include/cuda_runtime.h:1396:53: note: declared here static __CUDA_DEPRECATED __inline__ __host__ cudaError_t cudaBindTexture2D( ^ /usr/local/cuda-11.1/include/cuda_runtime.h: In instantiation of ‘cudaError_t cudaBindTexture2D(size_t*, const texture&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = float; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’: /home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:39:49: required from ‘kfusion::cuda::TextureBinder::TextureBinder(const A&, const texture&, const cudaChannelFormatDesc&) [with A = kfusion::cuda::PtrStepSz; T = float; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’ /home/robotking/yak_ros_test/src/yak/yak/src/cuda/tsdf_volume.cu:131:73: required from here /usr/local/cuda-11.1/include/cuda_runtime.h:1406:27: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const textureReference*, const void*, const cudaChannelFormatDesc*, size_t, size_t, size_t)’ is deprecated [-Wdeprecated-declarations] return ::cudaBindTexture2D(offset, &tex, devPtr, &desc, width, height, pitch); ^ /usr/local/cuda-11.1/include/cuda_runtime_api.h:7956:46: note: declared here extern __CUDA_DEPRECATED __host__ cudaError_t CUDARTAPI cudaBindTexture2D(size_t *offset, const struct textureReference *texref, const void *devPtr, const struct cudaChannelFormatDesc *desc, size_t width, size_t height, size_t pitch); ^ cd /home/robotking/yak_ros_test/build/yak; catkin build --get-env yak | catkin env -si /usr/bin/make --jobserver-fds=6,7 -j; cd - ............................................................................... Finished <<< yak [ 17.4 seconds ] _______________________________________________________________________________ Warnings << gl_depth_sim:make /home/robotking/yak_ros_test/logs/gl_depth_sim/build.make.000.log cc1: warning: command line option ‘-std=c++14’ is valid for C++/ObjC++ but not for C cd /home/robotking/yak_ros_test/build/gl_depth_sim; catkin build --get-env gl_depth_sim | catkin env -si /usr/bin/make --jobserver-fds=6,7 -j; cd - ............................................................................... Finished <<< gl_depth_sim [ 19.5 seconds ] [build] Summary: All 5 packages succeeded! [build] Ignored: None. [build] Warnings: 2 packages succeeded with warnings. [build] Abandoned: None. [build] Failed: None. [build] Runtime: 20.9 seconds total. `

WuRobotics commented 3 years ago

Hello @schornakj,

I tried to re-install the Ubuntu16.04 & CUDA & Nvidia driver multiple times with different versions but got no luck... Could you please help me to check this issue?

Thank you so much.

WuRobotics commented 3 years ago

Hi, @schornakj

I tried to use the Laptop with GTX 1050 GPU and accessories Nvidia Driver and CUDA, the Yak_ros demo is working normally. But the latest RTX 3070 still got no luck...

Since I hope to scan the object with high resolution, The GTX 1050 seems not good enough, could you please check the issue?

Thank you so much.

schornakj commented 3 years ago

Sorry for being quiet on this issue. I don't have hardware available to test newer GPUs myself, so I probably won't be able to directly help you resolve this issue.

I do not think there are many users of this project running on CUDA 11, so there could be changes to that library since CUDA 10 that could introduce some problems. A similar issue happened during the switch from CUDA 8 to CUDA 9.

If you're able to install Ubuntu 20.04 LTS on the computer with the RTX 3070 installed, I would be interested to know if that gives better results. Ubuntu 16.04 is quite old at this point, and newer versions of CUDA have some requirements about availability of certain versions of the GCC compiler and so on that can cause problems when trying to build these projects.

ros-industrial-attic / yak_ros