How to align the depth and RGB frame from time dimension

[seliber]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D455
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	Linux (Ubuntu 14/16/17)
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	imx8
SDK Version	2.50
Language	C++
Segment	Others

Issue Description

Recently, we found an issue about the alignment for depth and RGB image. the RGB frame was about 1 second later than the depth frame.

Do we need to set any configuration to align the depth and RGB image?

the API used:

_config.enable_stream(RS2_STREAM_COLOR, _colorWidth, _colorHeight, RS2_FORMAT_BGR8, 15);
_config.enable_stream(RS2_STREAM_DEPTH, _depthWidth, _depthHeight, RS2_FORMAT_Z16, 15);
_config.can_resolve(_pipeline);
_pipeline.start();
//use try_wait_for_frames() to get frame
auto frameSet;
_pipeline.try_wait_for_frames(&frameSet, 200);

[MartyG-RealSense]

Hi @seliber When using a cfg / config definition, it should be placed within the brackets of the pipeline start instruction, otherwise the program will not know it should be using your custom configuration and will ignore it and use the camera's default configuration instead.

_pipeline.start(_config);

[seliber]

Thanks @MartyG-RealSense , sorry, that's a typo. we use _pipeline.start(_config);

[MartyG-RealSense]

Thanks for the clarification. How does it behave if you do not put any instructions in the try_wait_for_frames bracket?

_pipeline.try_wait_for_frames();

[seliber]

then how can I get the frame result?

[MartyG-RealSense]

try_wait_for_frames is used to return a 'false' status if there is a timeout instead of allowing the program to throw an error and exit, as described at https://github.com/IntelRealSense/librealsense/issues/2422#issuecomment-423254709

[seliber]

actually , we monitor the result, it is always true. btw, do you mean to use _pipeline.wait_for_frames()?

[MartyG-RealSense]

Most people will use wait_for_frames() instead of try_wait_for_frames(), yes

[seliber]

Thanks, I'll try then.

[seliber]

@MartyG-RealSense Thanks for the advice, I have tried to change the API, but no lucky to get the correct frame. The RGB frame was still later than the depth frame.

[MartyG-RealSense]

https://github.com/IntelRealSense/librealsense/issues/1548#issuecomment-389070550 advises that there will be a temporal offset between depth and RGB frames but the wait_for_frames() instruction should be minimizing this by finding the best match between frame timestamps. This is assuming that frame drops are not occurring though.

You could try enforcing a constant FPS rate for both streams instead of permitting FPS to vary in order to test whether there is an FPS lag on the RGB stream. This can be done by having auto-exposure enabled and an RGB option called Auto-Exposure Priority disabled. C++ code for disabling auto-exposure priority on the RGB sensor is provided below.

auto sensor = pipe_profile.get_device().query_sensors()[1];
sensor.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, 0);

[seliber]

As you can see in the first post, we've fixed the both stream fps as 15.

And I also don't think it would related with the exposure, since it could be reproduced in many different light environemnt, But I'll try.

I can provide the raw data of rgb and depth, getting via this way if needed.

`auto processedFrameset = _pipeline.wait_for_frames(200); auto colorFrame = processedFrameset.get_color_frame(); auto depthFrame = processedFrameset.get_depth_frame(); //save the first frameset if(_isFirstFrame) { std::ofstream colorFile("color"); auto colorFileSize = _colorHeight _colorWidth _colorBytePerPixel; const char colorData = reinterpret_cast<const char >(colorFrame.get_data()); colorFile.write(colorData, colorFileSize); colorFile.close();

std::ofstream depthFile("depth");
auto depthFileSize = _depthHeight * _depthWidth *_depthBytePerPixel;
depthFile.write(reinterpret_cast<const char *>(depthFrame.get_data()), depthFileSize);
depthFile.close();

_isFirstFrame = false;

} `

[MartyG-RealSense]

When enabling depth and RGB at the same time, setting the two streams to the same value in the config instructions does not guarantee that they will both actually remain at that value unless the FPS is forced deliberately to remain constant, otherwise FPS of one of the streams (typically the RGB one) can drift to a slower value.

[seliber]

I tried to disabled auto-exposure, but still the two frames are not matched.

[MartyG-RealSense]

Auto-Exposure should be enabled. It is Auto-Exposure Priority, a different option, that should be disabled.

It may be useful to run the RealSense SDK's rs-align depth-color alignment example program to see whether there is a 1 second delay on the RGB on that example. If there is not a delay then it may suggest that the problem is in your script.

https://github.com/IntelRealSense/librealsense/tree/master/examples/align

If the examples were built when you installed librealsense then you should be able to find a pre-built executable version of rs-align in the /usr/local/bin folder.

[seliber]

I mean I use the way you told as below: auto sensor = pipe_profile.get_device().query_sensors()[1]; sensor.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, 0);

And I also get the original timestamp, the diff from RGB to Depth is large than 1000.

I'll enable BUILD_EXAMPLE and try again. Thanks for your help.

[seliber]

Sorry @MartyG-RealSense , we don't have UI output on the imx8 board, so that I cannot run "rs-align" example. But I read the source code of rs-align example, I didn't see too much different with the code in my side.

[seliber]

@MartyG-RealSense , just FYI, I re-compile the camera SDK with the option -DBUILD_NETWORK_DEVICE=true, then the RGB delay is back to normal.

but I didn't use any network features, just fetch the frame, why do I need to build with this option, it made the size of the libary increased

[MartyG-RealSense]

There is not an obvious reason why using the -DBUILD_NETWORK_DEVICE=true flag would have an effect on RGB on a system that is not using networking.

If you do not have UI output, does that mean that you are using a 'headless' setup without a display? If you are then I would recommend specifying in your librealsense build that it is a headless' system that does not require graphics support to be included in the build by using the CMake build terms -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=false

By setting DBUILD_GRAPHICAL_EXAMPLES=false then only text-based example programs and tools will be built, and OpenGL support and OpenGL-based graphical tools and examples such as realsense-viewer will be excluded.

[seliber]

cmake ../. -DCMAKE_BUILD_TYPE=Release -DFORCE_RSUSB_BACKEND=ON -DBUILD_NETWORK_DEVICE=true -DBUILD_GRAPHICAL_EXAMPLES=false -DBUILD_CV_EXAMPLES=false -DBUILD_CV_KINFU_EXAMPLE=false -DBUILD_GLSL_EXTENSIONS=false -DCMAKE_TOOLCHAIN_FILE=~/mycompiler.cmake

This is my compile option.

[MartyG-RealSense]

I cannot see anything in the above CMake build instruction that would clearly suggest why you are having problems with alignment. You could try a simpler build instruction though to eliminate the possibility that one of the flags is causing the problem, as some of the settings that were being set to false are already false by default anyway and I am not aware of a need to use DCMAKE_TOOLCHAIN_FILE unless building headless for the RealSense Android wrapper.

DBUILD_EXAMPLES is True by default, so if you want to exclude the headless textual tools then set it to False.

cmake ../. -DCMAKE_BUILD_TYPE=Release -DFORCE_RSUSB_BACKEND=ON -DBUILD_EXAMPLES=false -DBUILD_GRAPHICAL_EXAMPLES=false

[MartyG-RealSense]

Hi @seliber Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.