Better way to convert .bag file to images? different frame numbers obtained for rgb and depth when using rs-convert. Loss in frames.

IntelRealSense / librealsense

Intel® RealSense™ SDK

https://www.intelrealsense.com/

Apache License 2.0

7.6k stars 4.83k forks source link

Better way to convert .bag file to images? different frame numbers obtained for rgb and depth when using rs-convert. Loss in frames. #3689

Closed arake001 closed 5 years ago

arake001 commented 5 years ago

Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):
- Consider checking out SDK examples.
- Have you looked in our documentations?
- Is you question a frequently asked one?
- Try searching our GitHub Issues (open and closed) for a similar issue.
All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

Required Info
Camera Model	{ R200 / F200 / SR300 / ZR300 / D400 }
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	{Win (8.1/10) / Linux (Ubuntu 14/16/17) / MacOS
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC/Raspberry Pi/ NVIDIA Jetson / etc..
SDK Version	{ legacy / 2.<?>.<?> }
Language	{C/C#/labview/nodejs/opencv/pcl/python/unity }
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

MartyG-RealSense commented 5 years ago

The link below is an excellent discussion in February 2019 of this subject:

https://github.com/IntelRealSense/librealsense/issues/3208

arake001 commented 5 years ago

@MartyG-RealSense The rs_bag2image tool mentioned in the #3208 will that help in synchronizing the RGB and depth frames? I also got to know that it converts the depth data to gray-scale images. Is there another approach to get it back to the standard colored depth data?

MartyG-RealSense commented 5 years ago

@UnaNancyOwen would be the best person to ask about using color in rs,_bag2image. Hopefully now I have linked Una into this discussion, they can provide advice.

In regard to depth and color sync, I would recommend looking at the rs2::syncer instruction, which is discussed in the documentation link below.

https://github.com/IntelRealSense/librealsense/blob/master/doc/rs400_support.md#api-changes

arake001 commented 5 years ago

How can I sync the depth and RGB frames after using rs-convert? Can you provide the following steps to do that? @MartyG-RealSense

MartyG-RealSense commented 5 years ago

I do not have enough knowledge of this subject to be able to offer further advice, unfortunately. Hopefully another person on this forum can help. I do apologize.

arake001 commented 5 years ago

@dorodnic Can you help me with the issue of synchronizing the RGB and Depth frames? How can I use rs2::syncer after rs-convert? Or any other approach you would suggest me to synchronize?

arake001 commented 5 years ago

How to synchronize the depth and color images? I know there will be a drop in frames for sure. However, I want to see if I can synchronize the frames.

RealSenseCustomerSupport commented 5 years ago

Hi @ankith001, How about to use the timestamps and frame counters provided in the frame metadata to help to synchronize the color and depth? In addition, you can slightly modify the record-playback sample to record synchronize depth/color frames. https://github.com/IntelRealSense/librealsense/tree/master/examples/record-playback Thank you.

arake001 commented 5 years ago

I have long videos to synchronize. The videos are around 30-40 minutes long.

arake001 commented 5 years ago

@dorodnic @RealSense-Customer-Engineering @MartyG-RealSense @UnaNancyOwen What code do I need to use to get timestamps, frame number and convert bag files to png all at the same time?

arake001 commented 5 years ago

Hi @YangJiao1996 Did you get your code working for the rs-convert to synchronize the RGB and depth frames? Can you share it with us?

YangJiao1996 commented 5 years ago

Hi @ankith001, Unfortunately, you cannot synchronize the RGB, depth and/or IR frames after the .bag file is generated. Currently (after the 2.20 version) what both rs-convert and probably rs_bag2image can do is to extract the frame saved in the bag file.

That is to say, if you found out that your RGB and depth frames are not synchronized, it could the recording process that failed to synchronize streams from different sensors.

Dropping frame issues are common in D435 when I did all my experiments, and it makes the time synchronization a little bit tricky. I only worked with depth and IR frames. They all come from the same sensor, so it is easy to get synchronization using timestamps and frame numbers. In your case, my suggestion is to make use of the hardware trigger, timestamps and the frame number API from RealSense SDK, or accurate external timer for getting the exact timing information.

I hope that helps.

Yang

arake001 commented 5 years ago

Hi @YangJiao1996,

Thank you for your response. Any easy way to get the timestamps and frame number from the .bag file? Have you come across any code which does that?

YangJiao1996 commented 5 years ago

Hi @ankith001,

Please go through the API or example code for getting timestamps and/or frame numbers. Una's code is a great example to get started with.

One problem with using these to synchronize the streams is that both the timestamps and frame numbers are accumulated (from the time when sensors are activated), and you need to be aware of that.

Yang