search

NVIDIA-ISAAC-ROS / isaac_ros_visual_slam

Visual SLAM/odometry package based on NVIDIA-accelerated cuVSLAM
https://developer.nvidia.com/isaac-ros-gems
Apache License 2.0
862 stars 139 forks source link

Low frame rate #165

Open Cdfghglz opened 3 months ago

Cdfghglz commented 3 months ago

I am facing a strange issue when most of the time the node outputs 3-4 FPS while a realsense outputting image and other topics at ~90FPS.

image

I am running the stock ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense.launch.py.

The strange thing is, sometimes it just runs smoothly on ~90fps.

Its running on 2060 RTX, with no obvious difference in htop or nvidia-smi.

dslepichev commented 3 months ago

Hi Cdfghglz,

I can't reproduce your issue in the latest version.

  1. Get the latest version. The current cuVSLAM version is 12.2, but you use the old 11.4.
  2. Try to set enable_auto_exposure Troubleshooting Hardware Setup: Infra Camera Capped at low fps
  3. Try to upgrade firmware to RealSense lib/firmware see Isaac Ros RealSense setup It is recommended to use RealSense firmware version 5.15.1, librealsense version 2.55.1 and RealSense camera ROS driver version 4.51.1.
Cdfghglz commented 3 months ago

Hi dslepichev,

Thank you for your reply.

In the meantime I tried to update to isaac_ros 3.0.0 (cuVSLAM 12.2). Does the realsense version recommendation still hold? They have a pretty recent 4.55.1 ROS out recently along with newer firmware compatible with the 2.55.1.

Before I set all the right versions according to your advice, I will just add some more observations:

I run the node via docker, where the first time I enter the container, run source install/setup.bash, then launch the ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense.launch.py, most of the time it runs at ~90Hz. After stopping the node and running the same launch again, I observe the bug with 4Hz.

When I subscribe to the /camera/infra1/image_rect_raw topic on another machine (LAN network), the rate increases to about 60 :see_no_evil:

It seems like it cound have to do with timing in some way, therefore I checked the time stamps but observed nothing out of sync:

[INFO] [1718183427.890285895] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=889399808)"
[INFO] [1718183427.893684398] [my_subscriber]: inf 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=888936279)"
[INFO] [1718183427.894243831] [my_subscriber]: img 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=888936279)"
[INFO] [1718183427.894710841] [my_subscriber]: img 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=888936279)"
[INFO] [1718183427.895083151] [my_subscriber]: inf 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=888936279)"
[INFO] [1718183427.895508040] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=894401792)"
[INFO] [1718183427.900290013] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=899403776)"
[INFO] [1718183427.904920169] [my_subscriber]: inf 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=900057373)"
[INFO] [1718183427.905538024] [my_subscriber]: img 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=900057373)"
[INFO] [1718183427.906060085] [my_subscriber]: img 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=900057373)"
[INFO] [1718183427.906384078] [my_subscriber]: inf 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=900057373)"
[INFO] [1718183427.906685604] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=904405760)"
[INFO] [1718183427.910314977] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=909407744)"
[INFO] [1718183427.915403612] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=914409728)"
[INFO] [1718183427.916007566] [my_subscriber]: inf 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=911178711)"
[INFO] [1718183427.916562429] [my_subscriber]: img 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=911178711)"
[INFO] [1718183427.917065353] [my_subscriber]: img 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=911178711)"
[INFO] [1718183427.917454293] [my_subscriber]: inf 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=911178711)"
[INFO] [1718183427.920500239] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=919411712)"
[INFO] [1718183427.925511030] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=924413696)"
[INFO] [1718183427.927068601] [my_subscriber]: inf 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=922300781)"
[INFO] [1718183427.927618106] [my_subscriber]: img 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=922300781)"
[INFO] [1718183427.928082831] [my_subscriber]: img 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=922300781)"
[INFO] [1718183427.928387327] [my_subscriber]: inf 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=922300781)"
[INFO] [1718183427.930252145] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=929415168)"
[INFO] [1718183427.935267327] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=934417152)"
[INFO] [1718183427.938169068] [my_subscriber]: inf 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=933421875)"
[INFO] [1718183427.938706168] [my_subscriber]: img 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=933421875)"
[INFO] [1718183427.939161497] [my_subscriber]: img 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=933421875)"
[INFO] [1718183427.939483244] [my_subscriber]: inf 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=933421875)"
[INFO] [1718183427.940240003] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=939419136)"
[INFO] [1718183427.945247328] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=944421120)"
[INFO] [1718183427.949281337] [my_subscriber]: inf 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=944542969)"
[INFO] [1718183427.949828362] [my_subscriber]: img 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=944542969)"
[INFO] [1718183427.950313348] [my_subscriber]: img 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=944542969)"
[INFO] [1718183427.950613636] [my_subscriber]: inf 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=944542969)"
[INFO] [1718183427.951000246] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=949423104)"
[INFO] [1718183427.955236682] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=954424832)"
[INFO] [1718183427.960415073] [my_subscriber]: inf 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=955665039)"
[INFO] [1718183427.960957977] [my_subscriber]: img 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=955665039)"
[INFO] [1718183427.961443085] [my_subscriber]: img 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=955665039)"
[INFO] [1718183427.961768163] [my_subscriber]: inf 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=955665039)"
[INFO] [1718183427.962081974] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=959426816)"
[INFO] [1718183427.965360138] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=964428800)"
[INFO] [1718183427.970319782] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=969430784)"
[INFO] [1718183427.971676967] [my_subscriber]: inf 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=966786133)"
[INFO] [1718183427.972226444] [my_subscriber]: img 1 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=966786133)"
[INFO] [1718183427.972690452] [my_subscriber]: img 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=966786133)"
[INFO] [1718183427.973012945] [my_subscriber]: inf 2 Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=966786133)"
[INFO] [1718183427.975352092] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=974432256)"
[INFO] [1718183427.980298469] [my_subscriber]: imu Timestamp: "builtin_interfaces.msg.Time(sec=1718183427, nanosec=979435264)"

Anyways please just confirm the version recommendation and I will continue debugging.