Inconsistent frame queue when pulling frames from recorded bag file

[anguyen216]

Required Info
Camera Model	{D400 }
Firmware Version	(Open RealSense Viewer 2.36.0)
Operating System & Version	{Win (10)
Platform	PC/Raspberry Pi/ NVIDIA Jetson / etc..
SDK Version	{ legacy / 2.35.2.1937 }
Language	{python }
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

I observed an issue with inconsistent frame queue when pulling frames from recorded bag file. I'm looping through all frames from a bag file; at each frame, I align the depth and color frames (either depth --> color or color --> depth), and then save both frames into image files for visual inspection. To stop the loop, I determine the number of frames by using the following function that I wrote. The function simple extracts the length of the video (in second) and then multiplies the length with 30 (fps) to get the number of frames in the file

def get_num_frames(filename):
    """
    Extract the number of frames a bag file
    Input: filename - path to the bag file
    Output: number of frames in the bag file
    """
    cfg = rs.config()
    cfg.enable_device_from_file(filename)
    # setup pipeline for the bag file
    pipe = rs.pipeline()
    # start streaming from file
    profile = pipe.start(cfg)

    # setup playback
    playback = profile.get_device().as_playback()
    playback.set_real_time(False)
    # get the duration of the video
    t = playback.get_duration()
    # compute the number of frames (30fps setting)
    frame_counts = t.seconds * 30
    playback.pause()
    pipe.stop()
    return frame_counts

Function to align frames

def align_frame(frame, align_to):
    align = rs.align(align_to)
    aligned = align.process(frame)
    depth_frame = colorizer.colorize(aligned.get_depth_frame())
    color_frame = aligned.get_color_frame()
    dimg = np.asanyarray(depth_frame.get_data())
    cimg = np.asanyarray(color_frame.get_data())
    return dimg, cimg

I have two different approaches when it comes to aligning the frames and saving them. One is that I align frames as I loop through the frames set. The other is that I loop through the frames set, keep each composite frame and save them somewhere, and then align the frames in another loop after closing the pipeline. Please see the code snippet below for more details. The strange thing is, depends on my approach, the frames extraction result would be very different to each other and when compare to the playback in intel realsense viewer application. All approaches agree upon the length of the video. However, neither made it to the end of the video (when compare to the viewer application display)

First approach: align as looping through the frames

# get video length
frame_counts = get_num_frames(example_file)
print(frame_counts)
for i in range(frame_counts):
    print(i)
    frames = pipe.wait_for_frames()
    if not frames:
        print("Error: not all frames are extracted")
        break
    dimg, cimg = align_frame(frames, rs.stream.depth)
    io.imsave("./test_res3/" + str(i) + ".jpg", np.hstack((cimg, dimg)))
pipe.stop()

Second approach: loop through the bag file and save composite frames. Align frames after closing the pipeline

# get video length
frame_counts = get_num_frames(example_file)
print(frame_counts)
for i in range(frame_counts):
    print(i)
    frames = pipe.wait_for_frames()
    if not frames:
        print("Error: not all frames are extracted")
        break
    frames.keep()
    frameset.append(frames)
pipe.stop()

for i in range(len(frameset)):
    align_to = rs.stream.color
    dimg, cimg = align_frame(frameset[i], align_to)
    io.imsave("./test_res2/" + str(i) + ".jpg", np.hstack((cimg, dimg)))

Result of first approach: last frame extracted and aligned

Result of second approach: last frame extracted and aligned

According to viewer application, the last frame extracted by second approach is well before the last frame (see image below)

The correct last frame according to viewer application

Can someone please explain this behavior? This behavior is creating some troubles for me to create a consistent pipeline for my project. Thanks!

[MartyG-RealSense]

Hi @anguyen216 Rather than calculating the end of the bag based on length and FPS, a more accurate method to prevent the bag from looping back to the start after it reaches the final frame may be to add a repeat_playback=False condition:

cfg.enable_device_from_file(filename, repeat_playback=False)

Could you test please whether there is a positive difference to the number of frames extracted when using the above command?

[anguyen216]

Hi @MartyG-RealSense. This works regarding getting the same results as what is displayed by the viewer application. There's a positive difference to the number of frames extracted when using the above command when compared to the methods mentioned above. Additionally, even with the above method, if alignment is done at the same time within the initial loop, the result would be about 100 frames less from what is output by performing alignment after the pipeline is closed

[MartyG-RealSense]

I thought very carefully about your latest question and re-read the case from the start. As an alternative to doing the alignment and extraction in Python, I wonder if an alternative would be to use the SDK's rs-convert bag conversion tool to convert the bag to a ply point cloud file.

https://github.com/IntelRealSense/librealsense/tree/master/tools/convert

A member of the RealSense support team stated in another case that if a bag contains a depth and color stream, then the ply file extracted by rs-convert will be aligned.

https://support.intelrealsense.com/hc/en-us/community/posts/360033198613/comments/360008034794

The Windows version of the RealSense SDK has a pre-built executable version of rs-convert that can be launched from the Windows command prompt interface. Further details about this can be found in the link below:

https://github.com/IntelRealSense/librealsense/issues/7033#issuecomment-669161204

[anguyen216]

@MartyG-RealSense Thank you for looking carefully into this case and the useful suggestions. I'll definitely check out the rs-convert conversion tool. For the most part, though, the reason I'm using Python because I'm not too familiar with C++ to edit the source code of the conversion tool to for my application. But perhaps, I can use it as a baseline to compare and find out which approach in python would give me the most consistent frame extraction. Will keep you updated as I'm currently working on this and hopefully will have something more solid soon

[anguyen216]

@MartyG-RealSense Update: I just test the converter tool on one file. When converting bag file into .PLY format, I got the following result summary

100%
PLY converter
        589  frame(s) processed

When converting bag file into PNG format, I got the following result summary

PNG converter
        590 Infrared frame(s) processed
        590 Depth frame(s) processed
        304 Color frame(s) processed

Though the difference in the number of frames for depth and infrared mode aren't much difference between the two converter, number of RGB frames is only slightly more than half the number of depth and infrared when using PNG converter. Can you please explain these differences?

[MartyG-RealSense]

rs-convert has had a tendency in the past to drop some frames when converting to PNG. In a discussion about this, a RealSense team member offered a link to a Python script that they said they had successfully tested in extracting all depth, IR and color frames from a bag as images.

https://github.com/IntelRealSense/librealsense/issues/3500#issuecomment-486805956

The Python script:

https://gist.github.com/wngreene/835cda68ddd9c5416defce876a4d7dd9

[anguyen216]

@MartyG-RealSense Thank you for the resource. If it's not too much trouble for you, can you link the instructions on how to install the packages used in the scripts above, namely rosbag and cv_bridge , on macOS and Windows OS? I'm having a hard time looking for such instructions

[MartyG-RealSense]

Apologies for the delay in responding further. Accessing cv_bridge on Windows or MacOS does not sound like it is easy to implement. The link below was the best information that I could find.

https://answers.ros.org/question/302336/opencv-and-python-2-no-module-named-cv_bridge_boost-on-windows/

[anguyen216]

@MartyG-RealSense thank you for looking into this! After my response yesterday, I took a few more hours to dig into this and found instruction to install ROS on windows OS (no luck with macOS). However, I was not able to use the script above to extract frames from bag file as the version I installed was not exactly the same as the one used for the script above; the function calls and outputs changed quite a bit. In the interest of time, I decided to stick with pyrealsense2 library. Frames will still be dropped, but I can extract all information (depth and rgb aligned) in one go to avoid mismatching frames later on. Hopefully, once I have more time to look into ROS documentation, I'll be able to do this with more elegant codes. For now, I'm content :). Thank you again!

ROS installation instruction on windows OS http://wiki.ros.org/Installation/Windows

[MartyG-RealSense]

Great news that you found a satisfactory solution - thanks so much for the update, and good luck with your further work!

[MartyG-RealSense]

Case closed due to no further comments received.

[rajkundu]

I am working on trying to use rosbag files directly, as I am having some issues with pyrealsense2 and frame dropping. Here is what worked for me on macOS 10.15.7 Catalina using Python 3.8.5:

1) Install the following packages with homebrew: brew install lz4 brew install boost-python3

2) Install the following packages with pip (or pip3):

pip install py3rosmsgs
pip install --extra-index-url https://rospypi.github.io/simple/ rospy
pip install --extra-index-url https://rospypi.github.io/simple/ rosbag
pip install --extra-index-url https://rospypi.github.io/simple/ roslz4
pip install --extra-index-url https://rospypi.github.io/simple/ cv-bridge

However, I did a bunch of debugging, so if anything is not working, perhaps there is a dependency I installed somewhere along the way that I didn't catch. Let me know if you have any issues using the above.

[MartyG-RealSense]

Thanks very much @rajkundu for sharing your experience in this subject with the RealSense community! :)

[Esperer105]

good