what processing is done to generate depth/image_rect_raw?

[mjsobrep]

I'm having a tough time figuring out exactly what is done to generate depth/image_rect_raw.

It seems like the image is shifted so that the center matches the color center. Is that accurate? Is there also a rotational transform applied to make the depth image sensor plane the same as the color image sensor plane? Is anything else done?

[MartyG-RealSense]

Hi @mjsobrep The RealSense 400 Series data sheet document states: "The left and right imagers capture the scene and sends imager data to the depth imaging processor, which calculates depth values for each pixel in the image by correlating points on the left image to the right image and via shift between a point on the Left image and the Right image".

When depth to color alignment is enabled, the depth field of view size resizes to the color field of view size. This can mean that depth information on the outer regions of the image can be excluded if the camera model has a color sensor that has a smaller field of view than the depth sensor, such as on the D435 / D435i models.

[mjsobrep]

Hi @MartyG-RealSense, thanks for the quick reply. What happens when depth alignment is turned off (<arg name="align_depth" value="False"/>)? We still get a "rectified" image stream (/<cam name>/depth/image_rect_raw). What makes that image "rectified"? I assume that it has had any distortion removed (needed to do good depth matching). Has it also had the rotation and/or translation from depth to color removed? It is obvious that no re-scaling has been done.

We also receive the intrinsics for the color and depth via their camera info topics (/<cam name>/depth/camera_info and /<cam name>/color/camera_info). And we get the depth/color extrinsics from /<cam name>/extrinsics/depth_to_color. When I use these data to align the depth to the color, I find that including the translation in the extrinsics calculation seems to provide worse alignment. I can't tell if using the depth to color rotation matrix has much effect (for me it tends to be very close to the identity). I am trying to figure out if my numbers/math is off or if in fact that translation (and possibly rotation?) components of the extrinsics from depth to color have already been accounted for.

[MartyG-RealSense]

Rectification takes place at the point of capture in the Vision Processor D4 hardware inside the camera. Most stream profiles are rectified and have a form of Brown-Conrady distortion model applied. An exception to this is Y16 infrared, which is unrectified as it is used for camera calibration. I do not have information available about the precise processes that are applied during rectification.

The link below describes the functions that the vision processor performs.

https://www.stemmer-imaging.com/media/uploads/cameras/13/136603-Intel-RealSense-D4-Vision-Processor-Brochure.pdf

A RealSense user tried undistorting the images using the undistort function of OpenCV but found that it made little difference, as described in the link below.

https://support.intelrealsense.com/hc/en-us/community/posts/4405241971731/comments/4405458478099

As mentioned earlier in this discussion, when using depth-color alignment the RealSense SDK can use its align processing block to automatically adjust for differences between the depth and color sensors, such as the smaller field of view size of the RGB sensor on the D435 / D435i models and differences in the resolution of depth and color. Using your own method of alignment would mean that these automatic adjustments would not take place.

[mjsobrep]

Thanks @MartyG-RealSense this is a lot of helpful information!

[MartyG-RealSense]

You are very welcome, @mjsobrep - thanks very much for the update!