Camera Frame Drops when recording Rosbag

[makolele12]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
• [X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

I am using a ZED2i for SLAM. My SLAM algorithm is implemented in ROS and is really sensitive to sudden frame drops or changes in the frame rate. That is why I am expecting a stable frame rate from the camera.

As I am mainly doing offline tests, I am using the zed-ros-wrapper to publish the camera data and then recording it in rosbags. However, when analyzing the rosbags to see how stable the recordings were, I noticed there are multiple framedrops or frame rate changes during the recording.

I rewrote the bag's messages with header timestamps as explained here, as normally the rosbag's messages are timestamped with the time when the message was received and not with the time when the message was generated (the latter is what my SLAM algorithm expects).

I wrote the following Python Code to analyze the bag jittering and generate some plots:

import rosbag
import matplotlib.pyplot as plt
import numpy as np
import cv2

bag_path = '/path_to_rosbag/rosbag.bag'

bFirst = True
jitter_accel = []
ros_time_accel = []

with rosbag.Bag(bag_path) as inbag:
        print("-------------------- Input: %s ------------------------------" % bag_path)
        print(inbag)
        init_t = inbag.get_start_time()

        for topic, msg, t in inbag.read_messages():

            if topic == "/zed2i/zed_node/left_raw/image_raw_color":

                if bFirst:
                    init_t = t
                    last_t = t
                    bFirst = False
                    continue

                jitter_accel.append( (t - last_t).to_sec() * 1000.0 )
                ros_time_accel.append(t.to_sec()-init_t.to_sec())
                last_t = t

jitter_np = np.asarray(jitter_accel)
Hz_avg = 1000/jitter_np.mean()

plt.close('all')
plt.figure()
plt.title("Infra1")
plt.plot(ros_time_accel,jitter_accel)
plt.legend( [ str(round(Hz_avg,2)) + ' Hz'] )
plt.xlabel("Playback time [s]")
plt.ylabel("jitter [msecs]")
plt.show()

Basically, it plots the time difference delta_t between messaget_i and t_i+1 in a given camera topic.

Steps to Reproduce

1. Start the ZED Node with a ZED2i camera
2. Record a Rosbag
3. Analyze the resulting rosbag

Expected Result

As comparison I tested the same with an Intel RealSense D455 and its respective ros-wrapper. The result that I expect is shown in the plot below, a line around ~33ms (of course with some small peaks).

Actual Result

When initializing the ZED Node I used the following parameters:

pub_frame_rate = 30
grab_frame_rate = 30
resolution = 3 (VGA)
img_downsample_factor = 1.0

Therefore, I am expecting my plots to be a line somewhere around ~33.33 ms (30FPS). However, I get the following result for the left camera. The plot for the right camera looks really similar that is why i am not posting it.

One can see roughly a line around 33 ms, with multiple peaks in multiples of 33ms which are clearly frame drops. One can also see multiple peaks going below 30ms, which shows some messages being generated before expected.

I would like to know if there is a way to get a much stable result here.I do not believe the problem here is being caused by rosbag record because otherwise I would get similar results with the RealSense Camera. I guess the problem is from the zed-ros-wrapper itself.

Am I maybe misinterpreting the timestamps from the ZED camera? The timestamp that the zed-ros-wrapper writes to the message is the time when the image was taken right? Or does it have any other meaning?

ZED Camera model

ZED2i

Environment

OS: Ubuntu 20.04
CPU: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz x86_64
GPU: NVIDIA GeForce RTX 3080
ZED SDK version:  3.7.6
Other info: ROS Noetic

Anything else?

It is worth mentioning that:

• With rosbag record I am just recording the camera topic to avoid overhead.
• I disabled all the depth, position tracking, mapping and object detection modules from the ZED2i, as I do not need them for my purpose.
• I also tried to set grab_frame_rate = 100 while leaving pub_frame_rate = 30 and had similar results.
• I am not using any kind of compression on the images, rosbag record is subscribed to the RAW images topics.

[makolele12]

Update: when subscribing to the topics with gray images (instead of RGB), occurs the same problem.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[yingkunwu]

same issue