Using isaac_ros_visual_slam on a UAV

mzahana commented 1 year ago

Hi! I have the following setup

Intel Realsense D455 camera (firmware is updated), and using realsense-ros 4.51.1
Jetson Xavier NX with SSD and with 8GB RAM at mode (20W 6 CORE)
A quadcopter
The D455 is rigidly mounted on the quadcopter and facing front
I need to get the depth and color images for some object detection (using Yolov8) and localization (using depth), in addition to the infra1/2 for VSLAM

I managed to run the VSLAM with realsense on Jetson Xavier NX. With relatively slow motion (quadcopter is not flying, but I move it by hand for testing), the odom is fine. When I start to move the quadcopter (by hand) faster, I start to get Tracker lost frequently although I am moving in feature-rich environment .

*I get the following terminal output

[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-05-30-15-22-53-443574-ubuntu-30521
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [30532]
[INFO] [realsense2_camera_node-2]: process started with pid [30534]
[component_container-1] [INFO] [1685449374.589500580] [visual_slam_launch_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_visual_slam/lib/libvisual_slam_node.so
[realsense2_camera_node-2] [INFO] [1685449375.268323481] [camera.camera]: RealSense ROS v4.51.1
[realsense2_camera_node-2] [INFO] [1685449375.269560042] [camera.camera]: Built with LibRealSense v2.53.1
[realsense2_camera_node-2] [INFO] [1685449375.269683084] [camera.camera]: Running with LibRealSense v2.53.1
[component_container-1] [INFO] [1685449375.310327614] [visual_slam_launch_container]: Found class: rclcpp_components::NodeFactoryTemplate<isaac_ros::visual_slam::VisualSlamNode>
[component_container-1] [INFO] [1685449375.315995245] [visual_slam_launch_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<isaac_ros::visual_slam::VisualSlamNode>
[component_container-1] [INFO] [1685449375.774189843] [visual_slam_node]: cuVSLAM version: 11.0
[realsense2_camera_node-2] [INFO] [1685449376.351750606] [camera.camera]: Device with serial number 207322250752 was found.
[realsense2_camera_node-2] 
[realsense2_camera_node-2] [INFO] [1685449376.352935711] [camera.camera]: Device with physical ID 2-3.4-3 was found.
[realsense2_camera_node-2] [INFO] [1685449376.353058336] [camera.camera]: Device with name Intel RealSense D455 was found.
[realsense2_camera_node-2] [INFO] [1685449376.354654454] [camera.camera]: Device with port number 2-3.4 was found.
[realsense2_camera_node-2] [INFO] [1685449376.354764344] [camera.camera]: Device USB type: 3.2
[realsense2_camera_node-2] [INFO] [1685449376.463697372] [camera.camera]: getParameters...
[realsense2_camera_node-2] [INFO] [1685449376.482594849] [camera.camera]: JSON file is not provided
[realsense2_camera_node-2] [INFO] [1685449376.487211201] [camera.camera]: Device Name: Intel RealSense D455
[realsense2_camera_node-2] [INFO] [1685449376.489785605] [camera.camera]: Device Serial No: 207322250752
[realsense2_camera_node-2] [INFO] [1685449376.492092389] [camera.camera]: Device physical port: 2-3.4-3
[realsense2_camera_node-2] [INFO] [1685449376.494420645] [camera.camera]: Device FW version: 05.14.00.00
[realsense2_camera_node-2] [INFO] [1685449376.496469441] [camera.camera]: Device Product ID: 0x0B5C
[realsense2_camera_node-2] [INFO] [1685449376.498685600] [camera.camera]: Sync Mode: Off
[realsense2_camera_node-2] [WARN] [1685449381.825816596] [camera.camera]: re-enable the stream for the change to take effect.
[realsense2_camera_node-2] [WARN] [1685449384.081378417] [camera.camera]: re-enable the stream for the change to take effect.
[realsense2_camera_node-2] [INFO] [1685449384.297285215] [camera.camera]: Stopping Sensor: Stereo Module
[realsense2_camera_node-2] [INFO] [1685449384.800192530] [camera.camera]: Starting Sensor: Stereo Module
[realsense2_camera_node-2] [INFO] [1685449385.322523250] [camera.camera]: Open profile: stream_type: Infra(1), Format: Y8, Width: 640, Height: 360, FPS: 90
[realsense2_camera_node-2] [INFO] [1685449385.322735413] [camera.camera]: Open profile: stream_type: Infra(2), Format: Y8, Width: 640, Height: 360, FPS: 90
[realsense2_camera_node-2] [INFO] [1685449385.322942968] [camera.camera]: Stopping Sensor: Motion Module
[realsense2_camera_node-2] [INFO] [1685449385.432701192] [camera.camera]: Starting Sensor: Motion Module
[realsense2_camera_node-2] [INFO] [1685449385.548169766] [camera.camera]: Open profile: stream_type: Accel(0)Format: MOTION_XYZ32F, FPS: 200
[realsense2_camera_node-2] [INFO] [1685449385.548375017] [camera.camera]: Open profile: stream_type: Gyro(0)Format: MOTION_XYZ32F, FPS: 200
[realsense2_camera_node-2] [INFO] [1685449385.580145089] [camera.camera]: RealSense Node Is Up!
[realsense2_camera_node-2] [WARN] [1685449385.588349747] [camera.camera]: 
[realsense2_camera_node-2] [WARN] [1685449385.821399918] [camera.camera]: XXX Hardware Notification:Motion Module failure,1.68545e+12,Error,Hardware Error
[realsense2_camera_node-2] [WARN] [1685449385.821707218] [camera.camera]: Hardware Notification:Motion Module failure,1.68545e+12,Error,Hardware Error
[component_container-1] [INFO] [1685449387.060717654] [visual_slam_node]: Time taken by CUVSLAM_WarmUpGPU(): 11.286359
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/visual_slam_node' in container '/visual_slam_launch_container'
[component_container-1] [INFO] [1685449387.342580981] [visual_slam_node]: left_pose_right reading from CameraInfo
[component_container-1] [INFO] [1685449387.342735415] [visual_slam_node]: Baseline is : 0.095124
[component_container-1] [INFO] [1685449387.342809816] [visual_slam_node]: left_camera_pose_imu reading from TF Tree
[component_container-1] [INFO] [1685449387.343780742] [visual_slam_node]: Use use_gpu: true
[component_container-1] [INFO] [1685449387.343893831] [visual_slam_node]: Enable IMU Fusion: true
[component_container-1] [INFO] [1685449387.343958152] [visual_slam_node]: gyroscope_noise_density: 0.000244
[component_container-1] [INFO] [1685449387.344023945] [visual_slam_node]: gyroscope_random_walk: 0.000019
[component_container-1] [INFO] [1685449387.344077642] [visual_slam_node]: accelerometer_noise_density: 0.001862
[component_container-1] [INFO] [1685449387.344129611] [visual_slam_node]: accelerometer_random_walk: 0.003000
[component_container-1] [INFO] [1685449387.344195851] [visual_slam_node]: frequency: 200.000000
[component_container-1] [INFO] [1685449387.615445015] [visual_slam_node]: Time taken by CUVSLAM_CreateTracker(): 0.271186
[component_container-1] [WARN] [1685449389.411415293] [visual_slam_node]: Delta between current and previous frame [229.858560] is above threshold [22.000000]
[component_container-1] [WARN] [1685449389.422269267] [visual_slam_node]: Delta between current and previous frame [38.156288] is above threshold [22.000000]
[component_container-1] [WARN] [1685449389.431825016] [visual_slam_node]: Delta between current and previous frame [28.263680] is above threshold [22.000000]
[component_container-1] [WARN] [1685449389.569839535] [visual_slam_node]: Delta between current and previous frame [27.152384] is above threshold [22.000000]
[component_container-1] [WARN] [1685449389.788513539] [visual_slam_node]: Delta between current and previous frame [67.690752] is above threshold [22.000000]
[component_container-1] [WARN] [1685449389.817772856] [visual_slam_node]: Delta between current and previous frame [24.525312] is above threshold [22.000000]
[component_container-1] [WARN] [1685449389.830116419] [visual_slam_node]: Delta between current and previous frame [24.222208] is above threshold [22.000000]
[component_container-1] [WARN] [1685449390.146256477] [visual_slam_node]: Delta between current and previous frame [33.611264] is above threshold [22.000000]
[component_container-1] [WARN] [1685449390.202527176] [visual_slam_node]: Delta between current and previous frame [33.532416] is above threshold [22.000000]
[component_container-1] [WARN] [1685449390.345376835] [visual_slam_node]: Delta between current and previous frame [33.433856] is above threshold [22.000000]
[component_container-1] [WARN] [1685449390.820191314] [visual_slam_node]: Delta between current and previous frame [66.226432] is above threshold [22.000000]
[component_container-1] [WARN] [1685449390.869894338] [visual_slam_node]: Delta between current and previous frame [33.049088] is above threshold [22.000000]
[component_container-1] [WARN] [1685449391.274337123] [visual_slam_node]: Delta between current and previous frame [32.190720] is above threshold [22.000000]
[component_container-1] [WARN] [1685449391.322290747] [visual_slam_node]: Delta between current and previous frame [32.064768] is above threshold [22.000000]
[component_container-1] [WARN] [1685449391.526974158] [visual_slam_node]: Delta between current and previous frame [31.739648] is above threshold [22.000000]
[component_container-1] [WARN] [1685449391.951250754] [visual_slam_node]: Delta between current and previous frame [31.214848] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.096148216] [visual_slam_node]: Delta between current and previous frame [28.270848] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.128346166] [visual_slam_node]: Delta between current and previous frame [33.275904] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.285215283] [visual_slam_node]: Delta between current and previous frame [44.355328] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.350777695] [visual_slam_node]: Delta between current and previous frame [44.349184] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.430448398] [visual_slam_node]: Delta between current and previous frame [77.590784] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.453363115] [visual_slam_node]: Delta between current and previous frame [44.330496] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.486460054] [visual_slam_node]: Delta between current and previous frame [66.486016] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.567021585] [visual_slam_node]: Delta between current and previous frame [33.234944] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.699818465] [visual_slam_node]: Delta between current and previous frame [110.741760] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.707982482] [visual_slam_node]: Delta between current and previous frame [66.432512] is above threshold [22.000000]
[component_container-1] [WARN] [1685449392.968267910] [visual_slam_node]: Delta between current and previous frame [33.360128] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.110582457] [visual_slam_node]: Delta between current and previous frame [33.363456] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.155044705] [visual_slam_node]: Delta between current and previous frame [44.486144] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.264281322] [visual_slam_node]: Delta between current and previous frame [44.487680] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.287526092] [visual_slam_node]: Delta between current and previous frame [55.611904] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.599795057] [visual_slam_node]: Delta between current and previous frame [33.372160] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.629718991] [visual_slam_node]: Delta between current and previous frame [33.373184] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.641403537] [visual_slam_node]: Delta between current and previous frame [44.497920] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.662714968] [visual_slam_node]: Delta between current and previous frame [44.283136] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.823623180] [visual_slam_node]: Delta between current and previous frame [78.133248] is above threshold [22.000000]
[component_container-1] [WARN] [1685449393.833262386] [visual_slam_node]: Delta between current and previous frame [33.491712] is above threshold [22.000000]
[component_container-1] [WARN] [1685449394.704144367] [visual_slam_node]: Delta between current and previous frame [33.568000] is above threshold [22.000000]
[component_container-1] [WARN] [1685449394.800421445] [visual_slam_node]: Delta between current and previous frame [33.524992] is above threshold [22.000000]
[component_container-1] [WARN] [1685449394.888218629] [visual_slam_node]: Delta between current and previous frame [55.708928] is above threshold [22.000000]
[component_container-1] [WARN] [1685449394.944267245] [visual_slam_node]: Delta between current and previous frame [33.426944] is above threshold [22.000000]
[component_container-1] [WARN] [1685449395.175671826] [visual_slam_node]: Delta between current and previous frame [33.432320] is above threshold [22.000000]
[component_container-1] [WARN] [1685449395.703764228] [visual_slam_node]: Delta between current and previous frame [55.646208] is above threshold [22.000000]
[component_container-1] [WARN] [1685449395.984347538] [visual_slam_node]: Delta between current and previous frame [33.392640] is above threshold [22.000000]
[component_container-1] [WARN] [1685449396.295631466] [visual_slam_node]: Delta between current and previous frame [44.527616] is above threshold [22.000000]
[component_container-1] [WARN] [1685449396.509855553] [visual_slam_node]: Delta between current and previous frame [33.398528] is above threshold [22.000000]
[component_container-1] [WARN] [1685449396.516908611] [visual_slam_node]: Delta between current and previous frame [66.798336] is above threshold [22.000000]
[component_container-1] [WARN] [1685449396.659878623] [visual_slam_node]: Delta between current and previous frame [33.374464] is above threshold [22.000000]
[component_container-1] [WARN] [1685449396.733231927] [visual_slam_node]: Delta between current and previous frame [33.374976] is above threshold [22.000000]
[component_container-1] [WARN] [1685449397.565449854] [visual_slam_node]: Delta between current and previous frame [33.392384] is above threshold [22.000000]
[component_container-1] [WARN] [1685449397.779583604] [visual_slam_node]: Delta between current and previous frame [33.393152] is above threshold [22.000000]

So, what could be the issue, and how to solve it to get reliable odometry?

Thank you.

swapnesh-wani-nvidia commented 1 year ago

I am not sure if this is a problem anymore, however, it looks like when you are moving your uav faster with your hand, it causes motion blur which will result in Tracker is lost. Additionally, there are frame "drops" (warning messages showing that input images to Isaac ROS Visual Slam are not steady) and this will also result in bad tracking.

mzahana commented 1 year ago

Hi @swapnesh-wani-nvidia Thanks for your reply. I have replaced the Jetson Xavier NX with Orin NX. I am suing the same camera D455. Here is the output

ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense.launch.py 
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-08-21-07-13-37-588341-orin-nx-31595
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [31606]
[INFO] [realsense2_camera_node-2]: process started with pid [31608]
[component_container-1] [INFO] [1692602018.195577355] [visual_slam_launch_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_visual_slam/lib/libvisual_slam_node.so
[realsense2_camera_node-2] [INFO] [1692602018.210915211] [camera.camera]: RealSense ROS v4.51.1
[realsense2_camera_node-2] [INFO] [1692602018.211124594] [camera.camera]: Built with LibRealSense v2.54.1
[realsense2_camera_node-2] [INFO] [1692602018.211154387] [camera.camera]: Running with LibRealSense v2.54.1
[realsense2_camera_node-2] [INFO] [1692602018.466795712] [camera.camera]: Device with serial number 207322250752 was found.
[realsense2_camera_node-2] 
[realsense2_camera_node-2] [INFO] [1692602018.466957222] [camera.camera]: Device with physical ID 2-3-5 was found.
[realsense2_camera_node-2] [INFO] [1692602018.466980903] [camera.camera]: Device with name Intel RealSense D455 was found.
[realsense2_camera_node-2] [INFO] [1692602018.467621340] [camera.camera]: Device with port number 2-3 was found.
[realsense2_camera_node-2] [INFO] [1692602018.467672190] [camera.camera]: Device USB type: 3.2
[component_container-1] [INFO] [1692602018.476514724] [visual_slam_launch_container]: Found class: rclcpp_components::NodeFactoryTemplate<isaac_ros::visual_slam::VisualSlamNode>
[component_container-1] [INFO] [1692602018.476636169] [visual_slam_launch_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<isaac_ros::visual_slam::VisualSlamNode>
[realsense2_camera_node-2] [INFO] [1692602018.478754799] [camera.camera]: getParameters...
[realsense2_camera_node-2] [INFO] [1692602018.483103328] [camera.camera]: JSON file is not provided
[realsense2_camera_node-2] [INFO] [1692602018.484236806] [camera.camera]: Device Name: Intel RealSense D455
[realsense2_camera_node-2] [INFO] [1692602018.484276551] [camera.camera]: Device Serial No: 207322250752
[realsense2_camera_node-2] [INFO] [1692602018.484305608] [camera.camera]: Device physical port: 2-3-5
[realsense2_camera_node-2] [INFO] [1692602018.484321289] [camera.camera]: Device FW version: 5.14.0
[realsense2_camera_node-2] [INFO] [1692602018.484350314] [camera.camera]: Device Product ID: 0x0B5C
[realsense2_camera_node-2] [INFO] [1692602018.484363114] [camera.camera]: Sync Mode: Off
[component_container-1] [INFO] [1692602018.542016013] [visual_slam_node]: cuVSLAM version: 11.0
[realsense2_camera_node-2] [WARN] [1692602022.427257981] [camera.camera]: re-enable the stream for the change to take effect.
[realsense2_camera_node-2] [WARN] [1692602023.343521644] [camera.camera]: re-enable the stream for the change to take effect.
[realsense2_camera_node-2] [INFO] [1692602023.457865926] [camera.camera]: Stopping Sensor: Stereo Module
[realsense2_camera_node-2] [INFO] [1692602023.899850255] [camera.camera]: Starting Sensor: Stereo Module
[realsense2_camera_node-2] [INFO] [1692602024.128711399] [camera.camera]: Open profile: stream_type: Infra(1), Format: Y8, Width: 640, Height: 360, FPS: 90
[realsense2_camera_node-2] [INFO] [1692602024.128861036] [camera.camera]: Open profile: stream_type: Infra(2), Format: Y8, Width: 640, Height: 360, FPS: 90
[realsense2_camera_node-2] [INFO] [1692602024.128989136] [camera.camera]: Stopping Sensor: Motion Module
[realsense2_camera_node-2] [INFO] [1692602024.148161229] [camera.camera]: Starting Sensor: Motion Module
[realsense2_camera_node-2] [INFO] [1692602024.232241816] [camera.camera]: Open profile: stream_type: Accel(0)Format: MOTION_XYZ32F, FPS: 200
[realsense2_camera_node-2] [INFO] [1692602024.233334012] [camera.camera]: Open profile: stream_type: Gyro(0)Format: MOTION_XYZ32F, FPS: 200
[realsense2_camera_node-2] [WARN] [1692602024.242075038] [camera.camera]: 
[realsense2_camera_node-2] [INFO] [1692602024.254367767] [camera.camera]: RealSense Node Is Up!
[realsense2_camera_node-2] [WARN] [1692602024.904798919] [camera.camera]: XXX Hardware Notification:Motion Module failure,1.6926e+12,Error,Hardware Error
[realsense2_camera_node-2] [WARN] [1692602024.904963116] [camera.camera]: Hardware Notification:Motion Module failure,1.6926e+12,Error,Hardware Error
[component_container-1] [INFO] [1692602035.804148538] [visual_slam_node]: Time taken by CUVSLAM_WarmUpGPU(): 17.261979
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/visual_slam_node' in container '/visual_slam_launch_container'
[component_container-1] [INFO] [1692602035.816379790] [visual_slam_node]: left_pose_right reading from CameraInfo
[component_container-1] [INFO] [1692602035.816497778] [visual_slam_node]: Baseline is : 0.095124
[component_container-1] [INFO] [1692602035.816527315] [visual_slam_node]: left_camera_pose_imu reading from TF Tree
[component_container-1] [INFO] [1692602035.816700185] [visual_slam_node]: Use use_gpu: true
[component_container-1] [INFO] [1692602035.816723481] [visual_slam_node]: Enable IMU Fusion: true
[component_container-1] [INFO] [1692602035.816735290] [visual_slam_node]: gyroscope_noise_density: 0.000244
[component_container-1] [INFO] [1692602035.816749146] [visual_slam_node]: gyroscope_random_walk: 0.000019
[component_container-1] [INFO] [1692602035.816760283] [visual_slam_node]: accelerometer_noise_density: 0.001862
[component_container-1] [INFO] [1692602035.816770907] [visual_slam_node]: accelerometer_random_walk: 0.003000
[component_container-1] [INFO] [1692602035.816781051] [visual_slam_node]: frequency: 200.000000
[component_container-1] [INFO] [1692602035.840477898] [visual_slam_node]: Time taken by CUVSLAM_CreateTracker(): 0.023683
[component_container-1] [WARN] [1692602102.287425265] [visual_slam_node]: Delta between current and previous frame [55.637248] is above threshold [22.000000]
[component_container-1] [WARN] [1692602102.307967743] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602102.331423562] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602102.452048648] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602102.666873724] [visual_slam_node]: Delta between current and previous frame [66.765568] is above threshold [22.000000]
[component_container-1] [WARN] [1692602102.735556069] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602102.795024938] [visual_slam_node]: Delta between current and previous frame [44.510208] is above threshold [22.000000]
[component_container-1] [WARN] [1692602102.804026120] [visual_slam_node]: Delta between current and previous frame [44.509952] is above threshold [22.000000]
[component_container-1] [WARN] [1692602102.975146125] [visual_slam_node]: Delta between current and previous frame [100.148480] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.003259596] [visual_slam_node]: Delta between current and previous frame [66.764544] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.044393896] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.063819954] [visual_slam_node]: Delta between current and previous frame [33.382656] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.134840868] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.262656325] [visual_slam_node]: Delta between current and previous frame [33.382656] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.324321775] [visual_slam_node]: Delta between current and previous frame [44.511232] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.359634226] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.472112747] [visual_slam_node]: Delta between current and previous frame [33.382656] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.487359024] [visual_slam_node]: Delta between current and previous frame [55.637504] is above threshold [22.000000]
[component_container-1] [WARN] [1692602103.503826684] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602105.055418572] [visual_slam_node]: Delta between current and previous frame [111.275776] is above threshold [22.000000]
[component_container-1] [WARN] [1692602105.146670399] [visual_slam_node]: Delta between current and previous frame [155.784448] is above threshold [22.000000]
[component_container-1] [WARN] [1692602105.783245247] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602106.080468129] [visual_slam_node]: Delta between current and previous frame [122.402048] is above threshold [22.000000]
[component_container-1] [WARN] [1692602106.091965614] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602106.392043174] [visual_slam_node]: Delta between current and previous frame [66.765056] is above threshold [22.000000]
[component_container-1] [WARN] [1692602106.406191047] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602107.479466016] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602107.586975028] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602107.592642856] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602107.950217948] [visual_slam_node]: Delta between current and previous frame [222.550016] is above threshold [22.000000]
[component_container-1] [WARN] [1692602108.258539761] [visual_slam_node]: Delta between current and previous frame [111.275008] is above threshold [22.000000]
[component_container-1] [WARN] [1692602108.271172610] [visual_slam_node]: Delta between current and previous frame [44.509696] is above threshold [22.000000]
[component_container-1] [WARN] [1692602108.569183599] [visual_slam_node]: Delta between current and previous frame [55.638016] is above threshold [22.000000]
[component_container-1] [WARN] [1692602108.599987840] [visual_slam_node]: Delta between current and previous frame [44.509696] is above threshold [22.000000]
[component_container-1] [WARN] [1692602109.045607703] [visual_slam_node]: Delta between current and previous frame [77.892352] is above threshold [22.000000]
[component_container-1] [WARN] [1692602109.470589080] [visual_slam_node]: Delta between current and previous frame [33.382656] is above threshold [22.000000]
[component_container-1] [WARN] [1692602110.491127315] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602110.748886299] [visual_slam_node]: Delta between current and previous frame [33.382656] is above threshold [22.000000]
[component_container-1] [WARN] [1692602110.756979516] [visual_slam_node]: Delta between current and previous frame [55.636992] is above threshold [22.000000]
[component_container-1] [WARN] [1692602111.057656146] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602111.383013651] [visual_slam_node]: Delta between current and previous frame [33.382656] is above threshold [22.000000]
[component_container-1] [WARN] [1692602111.691683203] [visual_slam_node]: Delta between current and previous frame [44.509952] is above threshold [22.000000]
[component_container-1] [WARN] [1692602112.002517463] [visual_slam_node]: Delta between current and previous frame [100.147712] is above threshold [22.000000]
[component_container-1] [WARN] [1692602112.302540144] [visual_slam_node]: Delta between current and previous frame [44.510208] is above threshold [22.000000]
[component_container-1] [WARN] [1692602112.311156001] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602112.314272612] [visual_slam_node]: Delta between current and previous frame [33.382400] is above threshold [22.000000]
[component_container-1] [WARN] [1692602112.623021937] [visual_slam_node]: Delta between current and previous frame [55.637248] is above threshold [22.000000]
[component_container-1] [WARN] [1692602112.635166801] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602112.654919651] [visual_slam_node]: Delta between current and previous frame [33.382656] is above threshold [22.000000]
[component_container-1] [WARN] [1692602113.255506242] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602113.256288090] [visual_slam_node]: Delta between current and previous frame [55.638016] is above threshold [22.000000]
[component_container-1] [WARN] [1692602113.545871620] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602113.855534953] [visual_slam_node]: Delta between current and previous frame [44.509696] is above threshold [22.000000]
[component_container-1] [WARN] [1692602113.865750701] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602113.867405153] [visual_slam_node]: Delta between current and previous frame [33.382912] is above threshold [22.000000]
[component_container-1] [WARN] [1692602114.177402926] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602114.487118320] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602114.490468666] [visual_slam_node]: Delta between current and previous frame [44.510976] is above threshold [22.000000]
[component_container-1] [WARN] [1692602114.800155546] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602115.112020926] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602115.114585871] [visual_slam_node]: Delta between current and previous frame [100.147968] is above threshold [22.000000]
[component_container-1] [WARN] [1692602115.422399824] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602116.375076136] [visual_slam_node]: Delta between current and previous frame [55.638016] is above threshold [22.000000]
[component_container-1] [WARN] [1692602116.459297036] [visual_slam_node]: Delta between current and previous frame [44.509952] is above threshold [22.000000]
[component_container-1] [WARN] [1692602116.496063285] [visual_slam_node]: Delta between current and previous frame [33.360128] is above threshold [22.000000]
[component_container-1] [WARN] [1692602116.514504892] [visual_slam_node]: Delta between current and previous frame [44.512512] is above threshold [22.000000]
[component_container-1] [WARN] [1692602116.549682995] [visual_slam_node]: Delta between current and previous frame [44.512768] is above threshold [22.000000]
[component_container-1] [WARN] [1692602116.974023664] [visual_slam_node]: Delta between current and previous frame [44.513792] is above threshold [22.000000]
[component_container-1] [WARN] [1692602117.282149905] [visual_slam_node]: Delta between current and previous frame [144.667392] is above threshold [22.000000]
[component_container-1] [WARN] [1692602117.296088905] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602117.607043234] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602117.917772597] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602118.217007673] [visual_slam_node]: Delta between current and previous frame [55.640832] is above threshold [22.000000]
[component_container-1] [WARN] [1692602118.232370622] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602118.551892833] [visual_slam_node]: Delta between current and previous frame [66.768896] is above threshold [22.000000]
[component_container-1] [WARN] [1692602118.568844408] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602118.720360101] [visual_slam_node]: Delta between current and previous frame [44.513792] is above threshold [22.000000]
[component_container-1] [WARN] [1692602119.493896247] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602119.771192353] [visual_slam_node]: Delta between current and previous frame [855.874560] is above threshold [22.000000]
[component_container-1] [WARN] [1692602119.820164490] [visual_slam_node]: Delta between current and previous frame [300.322048] is above threshold [22.000000]
[component_container-1] [WARN] [1692602120.099730618] [visual_slam_node]: Delta between current and previous frame [300.740352] is above threshold [22.000000]
[component_container-1] [WARN] [1692602120.409006064] [visual_slam_node]: Delta between current and previous frame [301.826816] is above threshold [22.000000]
[component_container-1] [WARN] [1692602120.722135519] [visual_slam_node]: Delta between current and previous frame [299.601664] is above threshold [22.000000]
[component_container-1] [WARN] [1692602121.028452567] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602121.029293809] [visual_slam_node]: Delta between current and previous frame [301.417216] is above threshold [22.000000]
[component_container-1] [WARN] [1692602121.345375384] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602121.346111439] [visual_slam_node]: Delta between current and previous frame [300.516608] is above threshold [22.000000]
[component_container-1] [WARN] [1692602121.681898851] [visual_slam_node]: Visual tracking is lost
[component_container-1] [WARN] [1692602121.704800437] [visual_slam_node]: Delta between current and previous frame [300.184832] is above threshold [22.000000]
[component_container-1] [WARN] [1692602122.034775281] [visual_slam_node]: Delta between current and previous frame [300.884480] is above threshold [22.000000]
[component_container-1] [WARN] [1692602122.274707829] [visual_slam_node]: Delta between current and previous frame [300.019456] is above threshold [22.000000]

There is a lot of jittering, and I am not sure why. I am using the same launch file for realsense camera.

This happens with the camera hand-held and with slow speed.

I am not sure how to fix this issue.

swapnesh-wani-nvidia commented 1 year ago

Could you check if the o/p from RealSense node is coming at the desired rate of 90 fps? ros2 topic hz --window 100 <image_topic_name>

mzahana commented 1 year ago

Hi @swapnesh-wani-nvidia

Here is the output.

ros2 topic hz --window 100 /camera/infra1/image_rect_raw
WARNING: topic [/camera/infra1/image_rect_raw] does not appear to be published yet
average rate: 67.906
    min: 0.001s max: 0.032s std dev: 0.01376s window: 68
average rate: 75.018
    min: 0.001s max: 0.033s std dev: 0.01436s window: 100
average rate: 66.711
    min: 0.001s max: 0.033s std dev: 0.01518s window: 100
average rate: 46.834
    min: 0.001s max: 0.035s std dev: 0.01507s window: 100
average rate: 42.833
    min: 0.001s max: 0.035s std dev: 0.01444s window: 100
average rate: 78.792
    min: 0.001s max: 0.035s std dev: 0.01494s window: 100
average rate: 88.386
    min: 0.001s max: 0.034s std dev: 0.01432s window: 100
average rate: 88.377
    min: 0.001s max: 0.035s std dev: 0.01432s window: 100
average rate: 88.419
    min: 0.001s max: 0.033s std dev: 0.01445s window: 100
average rate: 88.282
    min: 0.001s max: 0.033s std dev: 0.01440s window: 100
average rate: 90.837
    min: 0.001s max: 0.033s std dev: 0.01425s window: 100
average rate: 90.711
    min: 0.001s max: 0.032s std dev: 0.01423s window: 100
average rate: 88.261
    min: 0.001s max: 0.033s std dev: 0.01423s window: 100
average rate: 90.780
    min: 0.001s max: 0.032s std dev: 0.01416s window: 100
average rate: 88.235
    min: 0.001s max: 0.033s std dev: 0.01430s window: 100
average rate: 90.883
    min: 0.001s max: 0.035s std dev: 0.01436s window: 100
average rate: 88.161
    min: 0.001s max: 0.036s std dev: 0.01465s window: 100
average rate: 88.210
    min: 0.001s max: 0.034s std dev: 0.01430s window: 100
average rate: 90.835
    min: 0.001s max: 0.034s std dev: 0.01436s window: 100
average rate: 90.856
    min: 0.001s max: 0.034s std dev: 0.01432s window: 100
average rate: 90.845
    min: 0.001s max: 0.035s std dev: 0.01422s window: 100
average rate: 90.786
    min: 0.001s max: 0.033s std dev: 0.01434s window: 100
average rate: 90.832
    min: 0.001s max: 0.032s std dev: 0.01438s window: 100
average rate: 88.207
    min: 0.001s max: 0.033s std dev: 0.01454s window: 100
average rate: 90.764
    min: 0.001s max: 0.033s std dev: 0.01428s window: 100

for the other image

ros2 topic hz --window 100 /camera/infra2/image_rect_raw
WARNING: topic [/camera/infra2/image_rect_raw] does not appear to be published yet
average rate: 38.718
    min: 0.001s max: 0.035s std dev: 0.01331s window: 40
average rate: 56.445
    min: 0.001s max: 0.035s std dev: 0.01567s window: 100
average rate: 63.738
    min: 0.001s max: 0.035s std dev: 0.01540s window: 100
average rate: 73.341
    min: 0.001s max: 0.035s std dev: 0.01479s window: 100
average rate: 88.232
    min: 0.001s max: 0.032s std dev: 0.01420s window: 100
average rate: 88.149
    min: 0.001s max: 0.033s std dev: 0.01438s window: 100
average rate: 88.376
    min: 0.001s max: 0.033s std dev: 0.01436s window: 100
average rate: 90.783
    min: 0.001s max: 0.032s std dev: 0.01418s window: 100
average rate: 88.230
    min: 0.001s max: 0.033s std dev: 0.01425s window: 100
average rate: 90.875
    min: 0.000s max: 0.035s std dev: 0.01423s window: 100
average rate: 88.177
    min: 0.000s max: 0.035s std dev: 0.01445s window: 100
average rate: 90.918
    min: 0.001s max: 0.034s std dev: 0.01424s window: 100
average rate: 88.111
    min: 0.001s max: 0.033s std dev: 0.01439s window: 100
average rate: 90.784
    min: 0.001s max: 0.035s std dev: 0.01428s window: 100
average rate: 88.511
    min: 0.001s max: 0.035s std dev: 0.01441s window: 100
average rate: 88.249
    min: 0.001s max: 0.033s std dev: 0.01445s window: 100
average rate: 88.294
    min: 0.001s max: 0.033s std dev: 0.01440s window: 100
average rate: 88.365
    min: 0.001s max: 0.033s std dev: 0.01432s window: 100
average rate: 90.722
    min: 0.001s max: 0.033s std dev: 0.01421s window: 100
average rate: 90.834
    min: 0.000s max: 0.034s std dev: 0.01428s window: 100
average rate: 90.809
    min: 0.001s max: 0.034s std dev: 0.01434s window: 100

Please note that I am also streaming RGB and depth images as I need them, and I am using USB C cable.

javieryu commented 1 year ago

I've encountered similar issues with my current setup Orin Nano DevKit + d455. Although, I haven't encountered tracking loss yet.

Various factors seem to trigger the dropped or slow frames warning message on my setup including subscribing to the landmarks pointcloud on a remote machine connected via Ethernet to the Jetson and subscribing to the realsense depth or color topics (locally or on remote).

furkansariyildiz commented 1 year ago

Hello,

Do you have any solution in this state?

mzahana commented 1 year ago

@furkansariyildiz unfortunately it didn't work well for me. The tracker can easily get lost even if I dampen the camera/imu.

I switched to open_vins package. It is much more stable, and I used it successfully in flight.

furkansariyildiz commented 1 year ago

@mzahana Thanks for your answer, I will look that package.

Best Regards

swapnesh-wani-nvidia commented 1 year ago

@mzahana, could you try with depth stream off along with emitter off? Leaving the IR projector on could result in "Tracker Lost" output. See the troubleshooting section 5(iv). I understand that you need depth images for other things and I am suggesting turning off the depth images to find the root cause.

mzahana commented 1 year ago

@swapnesh-wani-nvidia thanks for your reply. I already did that, and the issue is still there.

NVIDIA-ISAAC-ROS / isaac_ros_visual_slam

Using isaac_ros_visual_slam on a UAV #91