Color and Depth Maps Not Aligned as Speeds Increase

[GrainsofSalt]

Similar to https://github.com/IntelRealSense/librealsense/issues/4771

When in place, the depth and rgb are aligned perfectly. As the camera starts to move faster (in a straight line), the two images start to dis-align proportional to the vertical position (meaning at the bottom they are still aligned, but at the top they are completely dis-aligned). The exposure on the rgb was set to 25 microseconds so that the rgb images taken wouldn't blur when going around 5mph. I have been trying to understand how the camera initialization variables intermingle, but the documentation seems to be quite bare. For instance, I was trying to figure out if when auto exposure is disabled, does that mean exposure priority mode becomes useless? In any case, I enabled exposure priority and this did not help. Any suggestions as to how I can align the rgb and depth images when the rgb camera has manually set low exposure time?

[ArjanVroegop]

Hi,

Interesting question - at the moment I'm looking into the same issue. In my case I use a conveyor belt and I want to align the RGB and depth data of the objects passing the camera.

The settings I use:

• Color Camera --> Auto exposure off, Auto exposure priority off
• Depth Camera --> Inter cam sync mode 1, hardware trigger on

This way both cameras have the same frame rate (otherwise they vary a little bit). In my data I don see differences between top and bottom parts of the images, there seems to be a shift between depth and color which varies from time to time.

It would be helpful to get some information about how the camera works internally. Which camera are you using? I use a D435 and I think the color and depth cameras are not synced by hardware - this might be what explains this behaviour. I want to look into the D415 - I think this one has all sensors on a single chip so this might give better results. Does anyone have more info on this?

[MartyG-RealSense]

The RGB sensor on both the D415 and D435 has a rolling shutter, which is slower than the global shutter on the D435's IR imagers that produce the depth image.

The D415's IR imagers are rolling shutter like the RGB (on D415, both RGB and IR use the OmniVision OV2740 imagers component, whilst the D435 uses a different OmniVision IR imager component that is wider).

The link below may provide some useful explanation.

https://github.com/IntelRealSense/librealsense/issues/3554

The data sheet document for the 400 Series cameras is available here:

https://dev.intelrealsense.com/docs/intel-realsense-d400-series-product-family-datasheet

[MartyG-RealSense]

@ArjanVroegop You are correct, the RGB sensor is integrated into the circuit board of the D415. On the D435, the RGB sensor is attached separately via a cable.

In the caseless Depth Module board versions of 400 Series cameras, the D415 module is the only model with an RGB sensor.

[GrainsofSalt]

@ArjanVroegop I am using the D435

@MartyG-RealSense Basically you are saying that it will be impossible to get the two images to match? Inherently the rolling shutter will always take extra time to complete than a global shutter correct? Is there any way to circumvent this, or is this an unavoidable hardware flaw?

Thank you so much for the input guys

[MartyG-RealSense]

@GrainsofSalt I will post below some info an Intel RealSense team member posted on the Intel Support site's RealSense forum.

---

In general Depth and RGB sensors operate each with its own clock. In many use-cases though it is envisaged that the frame pairs will be temporally coherent to allow for inter-frame inference, such as point cloud and frames alignment (registration),especially when capturing dynamic scenes.

When D400 RGB and Depth sensors are set to the same FPS and the actual exposure does not exceed the frame period, then RGB sensor internally acts as Master and at the beginning of each frame it sends HW pulse to the Depth (Slave) sensor to start Depth generation as well. This results in Depth and RGB frames being captured simultaneously (timestamp offset < 1msec)

---

The full discussion is here:

https://support.intelrealsense.com/hc/en-us/community/posts/360037286634-Strange-FPS-when-enable-both-color-and-depth-streams

[GrainsofSalt]

@MartyG-RealSense I appreciate the help. This explains why the aligned image diverges as you move along the vertical axis. If the rgb acts as the master to initiate the depth sensor, they will be more aligned as the rolling shutter begins its action as opposed to when it is finishing. Well, thank you for the information!