Lidar points dropped in ros bag record

[yashmewada9618]

Hello,

I am trying to record a ros bag for Ouster lidar and an Intel Real sense camera including their IMU data and I am splitting the bag files every 3.2GB size margin. But after I do ros2 bag info <bag_file> I see that I have lost almost 800 messages for 92s of the bag file which is not good, even the IMU data is lost.

Everything is working perfectly offline without rosbag record and I am receiving data at the given Hz and perfect bandwidth. After following the changes from #302 the results were somewhat better but not perfect. I have also tried the steps from all the related issues here and they somewhat solved the problems I have been facing for over so long but didn't solve completely.

The lidar seems to drop messages as the number of subscribers increases. Has anyone faced this kind of issue or am I missing something here?

I've included the details below.

Thanks.

To Reproduce Ouster Lidar Launch

    ouster_launch = IncludeLaunchDescription(
        XMLLaunchDescriptionSource(
            [
                PathJoinSubstitution(
                    [FindPackageShare("ouster_ros"), "launch", "sensor.launch.xml"]
                )
            ]
        ),
        launch_arguments={
            "viz": "false",
            "sensor_hostname": "os-122212000733.local",  # Replace with the actual sensor hostname
            "lidar_mode": "1024x20",
            "udp_profile_lidar": "RNG19_RFL8_SIG16_NIR16",
            "metadata": "/home/scout/autonomy-stack/src/os-122212000733-metadata.json",  # Replace with relative path to metadata file
            "timestamp_mode": "TIME_FROM_ROS_TIME",
        }.items(),
    )

# No compression of bag file, just splitting.
    rosbag_record = ExecuteProcess(
        cmd=[
            "ros2",
            "bag",
            "record",
            "-o",
            directory,
            "--qos-profile-overrides-path",
            Qos_override,
            "--max-cache-size",
            "1000000000",  # 1 GB
            "--max-bag-size",
            "3221225472",  # 3Gb of raw data
            "--max-bag-duration",
            "1800",  # seconds
            "-s",
            "sqlite3",
            "-a",
        ],
        shell=True,
    )

Problem

[ros2-10]   /camera/camera/color/metadata: 329
[ros2-10]   /ouster/range_image: 219
[ros2-10]   /camera/camera/gyro/metadata: 2184
[ros2-10]   /ouster/nearir_image: 219
[ros2-10]   /ouster/reflec_image: 219
[ros2-10]   /ouster/signal_image: 219
[ros2-10]   /camera/camera/gyro/sample: 2184
[ros2-10]   /camera/camera/accel/metadata: 1110
[ros2-10]   /camera/camera/imu: 2184
[ros2-10]   /ouster/scan: 219
[ros2-10]   /camera/camera/color/image_raw: 329
[ros2-10]   /ouster/points: 727
[ros2-10]   /camera/camera/aligned_depth_to_color/image_raw: 327
[ros2-10]   /camera/camera/aligned_depth_to_color/camera_info: 327
[ros2-10]   /camera/camera/depth/image_rect_raw: 328
[ros2-10]   /camera/camera/depth/color/points: 327
[ros2-10]   /ouster/imu: 1935
[ros2-10]   /camera/camera/depth/camera_info: 328
[ros2-10]   /camera/camera/color/camera_info: 329
[ros2-10]   /camera/camera/accel/sample: 1110
[ros2-10]   /camera/camera/depth/metadata: 328
[ros2-10] Total lost: 15481

Platform (please complete the following information):

• Ouster Sensor: OS-0 32U
• Ouster Firmware Version: 2.5.3
• ROS version/distro: humble
• Operating System: Ubuntu 22.04
• Machine Architecture: x64
• git commit: 3f9790713efb56099a7348aec6bdefa3e6807279 (HEAD, origin/ros2)

[andre-nguyen]

Haven't tried this myself but I wonder what would happen if you used the composable node of rosbag recording instead of the standalone node.

Unfortunately I don't think this is available in ROS humble as it was only merged into rolling a few months ago. But it would allow you to skip the DDS layer and only use intra-process comms. AFAIK, symptons of driver performance issues manifest as missing points in a scan rather than missing entire scans.

[yashmewada9618]

Hello @andre-nguyen I tried something similar to composable nodes (if I understood their definition correctly) using a simple Python code that calls the subprocess of rosbag record but seems the recorded bag file still has some missing lidar scans.

There are missing lidar scans only in the recorded bag file and not in the offline node, so it seems either the rosbag is not able to record huge data (~300MB/s of bandwidth data from Ouster and Realsense) or the DDS layer is not able to transport these data at a high rate.

[andre-nguyen]

Can you try this PR https://github.com/ouster-lidar/ouster-ros/pull/302

I tried it and it worked for me in a preliminary test. So it sounds like it isn't a rosbag or a DDS issue.

[yashmewada9618]

Yes, the results I shared are from the PR you commented. I made the same changes in the humble branch for my setup. Another update regarding this is if I change the storage flag from sqlite3 to mcap I no longer see the total lost: messages while recording but the rosbag info says I lost around 5% of the messages. Recording just lidar data reduces this value to around 1%.

[Imaniac230]

Hi, @yashmewada9618, I've made some modifications and rebased the branch since your original post. Those were just stability fixes so I wouldn't expect a change in behavior, but still, have you tried from the latest state?

But, since you're saying that the driver is not dropping the packets, they probably might be getting lost somewhere further down the path. Also, judging by:

1. you are getting all the data when simply passing through to another subscriber (without any disk writing), but you're missing them in a stored bag file (or when writing to disk),
2. you're getting better results by simply changing the storage format,

I would suspect a problem at the recording/writing side. Have you checked if your disk usage and write speeds are reaching expected rates while the bag is being recorded? Is the bag still missing data if you lower the data rate further with a different lidar mode or lidar packet profile?

I didn't perform any tests with bag recording, so I might try and see how the driver behaves.