Does the D405 require alignment between RGB and depth streams?

[xEnVrE]

Required Info
Camera Model	D405
Firmware Version	05.12.14.100
Operating System & Version	Ubuntu 20.04
Kernel Version (Linux Only)	6.2.7
Platform	PC
SDK Version	2.53.1
Language	C++
Segment	Others

Issue Description

I made some experiments using the rs-align example in C++. I have modified it so that I can run it using the native resolution of my application, i.e. 640x480, and I can disable the alignment completely if needed.

It seems that using a D405 camera the software-based alignment is not working as expected.

I am attaching a video which shows the behavior.

https://user-images.githubusercontent.com/6014499/237053011-6b6dcb73-2b99-4ee2-91fd-41ed25ebb0d6.mp4

As you can see, when I enable the alignment the depth and RGB streams get not aligned and the depth is kinda zoomed. Is such behavior of the rs2::align software block normal? Do I need to use alignment at all with the D405 camera?

Thank you

cc @MartyG-RealSense

[MartyG-RealSense]

Hi @xEnVrE The D405 camera model does not have a separate RGB sensor component and RGB is instead provided by the depth sensor. This means that RGB is already in alignment with the left IR sensor that depth originates from.

It is fine to use D405 with depth-color alignment scripts such as rs-align for C++ and align_depth2color.py for Python to create an aligned image of both depth and RGB data,

D405 uses a different depth unit scale scaling factor to other 400 Series models. Most models have a default depth scale of 0.001, whilst the default scale of D405 is 0.01.

Does the alignment occur correctly without your resolution modification, please?

---

I note that your firmware version at the top of this case is listed as 5.12.14.100. The firmware version that must be used with SDK 2.53.1 is 5.14.0.0.

Kernel version 6.2.7 is not supported by the RealSense SDK. The newest supported kernel at the time of writing this is 5.15. The SDK can be used with an unsupported kernel but there may be unpredictable consequnences in regards to stability. If you are unable to change your kernel version then you can build the SDK from source code with the RSUSB backend installation method so that the kernel is bypassed and the SDK does not need to be kernel-patched.

[xEnVrE]

Hi @MartyG-RealSense thank you for your the prompt answer.

This means that RGB is already in alignment with the left IR sensor that depth originates from

Thanks, then we can avoid using the alignment software-side.

D405 uses a different depth unit scale scaling factor to other 400 Series models. Most models have a default depth scale of 0.001, whilst the default scale of D405 is 0.01.

We are aware of it however we did not change any scale within the rs-align executable.

I note that your firmware version at the top of this case is listed as 5.12.14.100. The firmware version that must be used with SDK 2.53.1 is 5.14.0.0.

Kernel version 6.2.7 is not supported by the RealSense SDK. The newest supported kernel at the time of writing this is 5.15. The SDK can be used with an unsupported kernel but there may be unpredictable consequnences in regards to stability. If you are unable to change your kernel version then you can build the SDK from source code with the RSUSB backend installation method so that the kernel is bypassed and the SDK does not need to be kernel-patched.

To avoid any confusion we moved to another setup with Kernel 5.11.0-46-generic and we compiled using RSUSB in any case to avoid any issue. I wanted to also clarify that we are compiling with CUDA enabled.

Does the alignment occur correctly without your resolution modification, please?

We repeated our tests.

The original rs-align executable is running at 1280x720 and seems fine in terms of calibration with the D405.

However, I think - please correct me if I am wrong - that the software-based alignment should work as well with a different resolution. Then we changed these lines

https://github.com/IntelRealSense/librealsense/blob/b874e42685aed1269bc57a2fe5bf14946deb6ede/examples/align/rs-align.cpp#L40-L52

to

    const std::size_t width = 640;
    const std::size_t height = 480;

    // Create and initialize GUI related objects
    window app(width, height, "RealSense Align Example"); // Simple window handling
    ImGui_ImplGlfw_Init(app, false);      // ImGui library intializition
    rs2::colorizer c;                     // Helper to colorize depth images
    texture depth_image, color_image;     // Helpers for renderig images

    // Create a pipeline to easily configure and start the camera
    rs2::pipeline pipe;
    rs2::config cfg;
    if (!serial.empty())
        cfg.enable_device(serial);
    cfg.enable_stream(RS2_STREAM_DEPTH, width, height);
    cfg.enable_stream(RS2_STREAM_COLOR, width, height);
    pipe.start(cfg);

in order to test it with the resolution we are interested in, i.e., 640 x 480.

We got this result:

where the alignment is - according to us - not correct.

Indeed, by modifying the rs-align executable such that it does not perform the alignment, we got the following:

Thank you for your help!

[MartyG-RealSense]

You are very welcome! Can you confirm whether modifying rs-align to not perform alignment resolves your problem, please?

[xEnVrE]

Can you confirm whether modifying rs-align to not perform alignment resolves your problem

It seems so @MartyG-RealSense.

However, I was just wondering why using the rs2::align, as done inside the rs-align executable, breaks the alignment with this resolution. Can you confirm whether this is the expected behavior or not?

Thank you

[MartyG-RealSense]

I believe that you are the first person to report an issue with the C++ based rs-align example when using D405, as D405 users have always used the Python-based equivalent example align_depth2color.py example up until now. rs-align and align_depth2color.py both perform alignment using the align_to instruction.

I have just tested the Windows version of rs-align with a D405 and not seen misalignment though, but was using the pre-built executable version and so had not modified the resolution.

[xEnVrE]

but was using the pre-built executable version and so had not modified the resolution.

Indeed, with the pre-built executable we also don't have problem. However It would be great if it worked also with other resolutions :)

Thanks

[MartyG-RealSense]

Which cfg stream configuration instructions are you using to define 640x480 resolution in your rs-align custom version? Is it something like this:

cfg.enable_stream(RS2_STREAM_DEPTH, 640, 480, RS2_FORMAT_Z16, 30)
cfg.enable_stream(RS2_STREAM_COLOR, 640, 480, RS2_FORMAT_RGB8, 30)

[xEnVrE]

Posting here the first lines of the custom version of rs-align:

    std::string serial;
    if (!device_with_streams({ RS2_STREAM_COLOR,RS2_STREAM_DEPTH }, serial))
        return EXIT_SUCCESS;

    const std::size_t width = 640;
    const std::size_t height = 480;
    // Create and initialize GUI related objects                                                                                                                                               
    window app(width, height, "RealSense Align Example"); // Simple window handling                                                                                                            
    ImGui_ImplGlfw_Init(app, false);      // ImGui library intializition                                                                                                                       
    rs2::colorizer c;                     // Helper to colorize depth images                                                                                                                   
    texture depth_image, color_image;     // Helpers for renderig images                                                                                                                       

    // Create a pipeline to easily configure and start the camera                                                                                                                              
    rs2::pipeline pipe;
    rs2::config cfg;
    if (!serial.empty())
        cfg.enable_device(serial);
    cfg.enable_stream(RS2_STREAM_DEPTH, width, height, RS2_FORMAT_Z16, 30);
    cfg.enable_stream(RS2_STREAM_COLOR, width, height, RS2_FORMAT_RGB8, 30);
    pipe.start(cfg);

In summary, yes it is the same as you posted.

[MartyG-RealSense]

Is it still misaligned if you change the window size back to 1280x270 instead of using 640x480 but still keep the streams at 640x480

window app(1280, 720, "RealSense Align Example"); // Simple window handling

[MartyG-RealSense]

Hi @xEnVrE Do you require further assistance with this case, please? Thanks!

[xEnVrE]

Hi @MartyG-RealSense, we kind of solved by disabling the alignment.

However the problem still persists for those who want to use the alignment for whatever reason.

I will try to perform the last test you asked today, sorry for the delay.

[MartyG-RealSense]

That's no problem at all. I look forward to your next report. Thanks very much for the update!

[xEnVrE]

Is it still misaligned if you change the window size back to 1280x270 instead of using 640x480 but still keep the streams at 640x480

We made this test, it seems that the problem persists:

With software-based alignment

Without software-based alignment

[MartyG-RealSense]

The above images do not look as though they are misaligned. The black silhouette around the object is a common effect that occurs when depth and color are mapped together. It is called occlusion. The occlusion in this image is actually good quality, as the silhouette surrounds the object evenly on all sides instead of the shadow being off to one side of the object in particular.

[xEnVrE]

Dear @MartyG-RealSense thank you for your comment.

Unfortunately, I cannot agree with you on the fact that both images are not misaligned.

To me, if you consider the second image, it is clear that the alignment is very good. For example, the boundaries of the cup match when comparing RGB and depth:

Instead, if you take the first image - where the librealsense SDK is processing the depth to align it with the RGB image - it is clear to me that the alignment is quite off. The red arrow point at where the cup ends in the RGB image, while the green arrow points where the cup ends in the depth image.

Please, let me know what do you think! Thank you

[MartyG-RealSense]

Thanks very much for the clarification. It looks though as though your workaround for using rs-align with D405 without alignment is going to be the best approach for using a custom resolution.

[MartyG-RealSense]

Hi @xEnVrE Do you require further assistance with this case, please? Thanks!

[xEnVrE]

I would say no, we made the tests you asked for and we were waiting for a fix from your side.

As you stated, one solution is to disable alignment.

However I think it should be stated somewhere that the software-based alignment is broken for some resolutions with the D405 so that other people are aware of it.

Thanks

[MartyG-RealSense]

The alignment on rs-align became incorrect for you when a custom resolution of 640x480 was set and worked normally when using its default 1280x720 resolution. So the example program is working as intended when the program is unmodified.

I understand your concern though in regard to your wish to be able to customize the resolution used with alignment. It is certainly an issue that can be returned to at a future date if others experience a similar issue with their D405.

[xEnVrE]

So the example program is working as intended when the program is unmodified.

For sure. However the problem is not with the specific executable per se, rather on the alignment procedure that your SDK offers. The fact that rs-align has a default resolution of 1280x720 is, at least in my opinion, just a coincidence. Afterall, rs-align is just an example, as you stated.

This issue is more about the alignment of the D405 in general rather than the rs-align executable though. Sorry for being picky here.

[MartyG-RealSense]

It's no trouble at all. I have highlighted your issue to my Intel RealSense colleagues to seek their advice regarding whether a custom resolution can be set with D405.

[MartyG-RealSense]

My Intel RealSense colleagues responded that this issue with D405 mis-alignment is known to them and an internal report to look at the problem has already been created.

In the meantime they suggested using 640x360 or 848x480 resolution instead of 640x480, though this may not be an option for you as the rest of your application relies on 640x480.

[xEnVrE]

Hi @MartyG-RealSense,

Thank you for the update!

We could actually use those other resolutions if the alignment works with them. Moving from 640x480 to 640x360 should not be a big deal for us.

Thanks again for the update. Looking forward towards a final resolution for the issue.

[MartyG-RealSense]

Hi @xEnVrE Were you able to make use of 640x360 resolution to provide correct alignment, please? Thanks!

[xEnVrE]

Hi @MartyG-RealSense, unfortunately I do not have access to the camera setup these days. If possible, I will update the issue with the tests once ready.

Thank you

[MartyG-RealSense]

Thanks very much @xEnVrE for the update.

[Daa98]

Unfortunately, I also found that infrared 1 is not aligned to color well.

[MartyG-RealSense]

Hi @Daa98 The RealSense Viewer tool does not support alignment in its 2D mode (shown above). It only maps two different streams together in its 3D pointcloud mode.

On the D405 camera model, RGB color is provided by the left infrared sensor, so the left infrared and RGB images have the same viewpoint. The two images are not having alignment performed on them.

[Daa98]

Hello, @MartyG-RealSense Um, what I meant was that in theory, the near-infrared and RGB should be perfectly aligned (because they come from the same camera). However, in both the RealSense Viewer tool and in the code, these two images are not perfectly aligned at a resolution of 1280x720. This misalignment is causing issues with my own stereo matching algorithm using the two IR images, and the resulting depth map cannot be aligned completely with the RGB image. Do you have any solutions or suggestions to address this issue? Thank you for your reply!

[MartyG-RealSense]

Hi @Daa98 I can see the misalignment in your RGB and Infrared 1 images above. I performed repeated tests and observed a very slight misalignment in my own camera on each test.

Perfect alignment between depth and between RGB color from the left infrared sensor is promised in point number 2 of the section of Intel's camera tuning paper at the link below.

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#use-the-left-color-camera

The D405 is a unique situation though, as it is the only RealSense model with built-in RGB that obtains it from the depth sensor instead of a separate RGB sensor and so the same match-up promise cannot be guaranteed for Infrared 1 and RGB. The RGB image obtained from the depth sensor is also passed though an Image Signal Processor (ISP) chip in the camera's hardware to produce the final image, and this adjustment cannot be bypassed.

[Daa98]

Hello, @MartyG-RealSense . Thanks. So bad to hear that. Misalignment is more serious for close range objects, and the D405 is specifically designd for close object.

[MartyG-RealSense]

Alignment between RGB and infrared is not usually an issue with RealSense cameras, as alignment between those stream types is not supported. One of the two stream types has to be Depth (depth to color, or depth to infrared).

The only available solutions for color to infrared alignment are to follow the complicated advice at https://github.com/IntelRealSense/librealsense/issues/1556#issuecomment-389061477 or to align three streams including depth using the method at https://github.com/IntelRealSense/librealsense/issues/5093

[MartyG-RealSense]

Hi @xEnVrE Do you require further assistance with this case, please? Thanks!

[xEnVrE]

At the moment we are happy with the solution consisting of disabling the software alignment. Any future software update will be welcome of course. Thanks for the support thus far!

[MartyG-RealSense]

You are very welcome! As you do not require further assistance right now, I will close this issue. Please feel free to re-open it at a future date or create a new issue. Thanks again!