isaac_ros_visual_slam as simulation return [CUDA] failure

[muratkoc503]

Hi, I am working with isaac_ros_visual_slam package and Gazebo.

My error: [component_container-1] [INFO] [1662379434.366503061] [visual_slam_node]: Use use_gpu: true [component_container-1] [ERROR] 140409917632320 [CUDA] failure: 13, file: /home/akorovko/Code/elbrus/src/modules/cuda_modules/src/lk_tracker.cpp, line: 9

Also, i use pytorch environment with cuda.

Thanks for reply.

[muratkoc503]

I work on Ros Galactic. Maybe this is reason of my error. i dont know.

[hemalshahNV]

Could you describe the platform you're running on? Are you running within the Isaac ROS Dev base Docker container or natively on host? It looks like an x86 machine running ROS2 Humble (not Galactic which is incompatible) but with which NVIDIA GPU available for CUDA?

[muratkoc503]

My knowledge is not good at docker. So, sorry. I work on 64 bit on computer.(laptop). I run to isaac_ros_visual_slam on Docker container, but i run to Gazebo simulator on natively . CUDA Version: 11.7. I will look tomorrow. I use Ros2 Galactic on laptop. Thanks for Humble case. nvidia-smi:

[muratkoc503]

I try to run rosbag example which is "small_pool_test". But, it give same error. My GPU is NVIDIA GeForce MX110. Is it enough for isaac_ros_visual_slam package? Also, i try path as "nvidia-container-runtime" and "/usr/build/nvidia-container-runtime", but this is not work. I install nvidia-container-runtime package is seen belove: I use Ubuntu focal(20.04) and jammy(22.04). But bionic version is installed. I dont know, what is the reason of my error. I doubt from my nvidia-container-runtime. Also i try Ros2 Galactic and Humble.

[muratkoc503]

Also, i install nvidia-container-runtime to native computer. I dont install nvidia-container-runtime to this(isaac_ros_visual_slam) Docker container. Does the nvidia-container-runtime installing in the Docker container is necessary?

[hemalshahNV]

It appears you're running this on a Maxwell architecture GPU. We think this may not be recent enough for Isaac ROS Visual SLAM to work with. Can you try running on a Volta or newer GPU? We'll work on a more graceful failure message if this is indeed the problem you're running into. With only 2GB of memory, running with Gazebo may not be leaving enough video memory for Isaac ROS Visual SLAM. Can you check how much video memory is left over once Gazebo is running?

[muratkoc503]

I am not working with Gazebo, i am running bag file now:

source /workspaces/isaac_ros-dev/install/setup.bash && \ ros2 bag play src/isaac_ros_visual_slam/isaac_ros_visual_slam/test/test_cases/rosbags/small_pol_test/

Also, My Gazebo doesn't use GPU. Probably i dont use GPU correctly.

watch nvidia-smi: (on natural computer)

watch nvidia-smi: (in docker container)

top: (in docker container)

component container, rviz2 and all other program doesn use GPU. Only X-server and office program use GPU.

Also, i have 2GB memory. But only 3MB is used. Gazebo doesn't use GPU.

[muratkoc503]

I think, my installation is correct. Because i test my GPU(Nvidia Container) at my native computer and any docker container.

Run GPU in native computer: (BlackScholes sample)

Run GPU in docker container: (ros_caffe sample)

Also, i run optional command after installing isaac_ros_visual_slam package: colcon test --executor sequential

This command get below error:

Starting >>> isaac_ros_visual_slam
--- stderr: isaac_ros_visual_slam                         
Errors while running CTest
Output from these tests are in: /workspaces/isaac_ros-dev/build/isaac_ros_visual_slam/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Finished <<< isaac_ros_visual_slam [18.2s]  [ with test failures ]

So, i read log file. i see 3 errror message in : colcon test --executor sequential command

[component_container-1] [ERROR] 140288154709824 [CUDA] failure: 13, file: /home/akorovko/Code/elbrus/src/modules/cuda_modules/src/lk_tracker.cpp, line: 9
[ERROR] [component_container-1]: process has died [pid 44573, exit code -6, cmd '/opt/ros/humble/install/lib/rclcpp_components/component_container --ros-args -r __node:=visual_slam_container -r __ns:=/'].
[ERROR] 140288154709824 [CUDA] failure: 13, file: /home/akorovko/Code/elbrus/src/modules/cuda_modules/src/lk_tracker.cpp, line: 9

Also, i didn't can't try with Volta or new GPU. I have only this computer. Probably, my computer doesn't support this. I don't know. I have 2GB GPU memory. Also, this process (isaac_ros_visual_slam.launch.py with GPU ) run ~5 second. After i see [CUDA] failure: 13,... error. I see to run nvidia-container with watch nvidia-smi command as 5 second. After shutdown. (After installation, only first times, not 2,3. times)

[muratkoc503]

I try to Ros BAG sample in isaac_ros_nvblox package (nvblox_pol sample). It work correct for me.

Result: Run GPU isaac_ros_nvblox:

Rviz Result:

Probably, [CUDA] failure: 13, file: /home/akorovko/Code/elbrus/src/modules/cuda_modules/src/lk_tracker.cpp, line: 9 error interested in isaac_ros_visual_slam package. isaac_ros_nvblox work my computer with GPU.

[jaiveersinghNV]

Isaac ROS Visual SLAM is only compatible with Volta and newer GPU architectures.

[madgrizzle]

Can someone explain why a Volta (or newer) GPU is required? I've installed and have successfully run this on a Tesla P4 that is based on Pascal. I've yet to incorporate it into my robot, but what specifically does the Volta architecture provide that's missing on the Pascal that's in use by visual slam?