Infrared Aligned to Color

[tensorfreitas]

Required Info
Camera Model	R200 and F200
Firmware Version	1.0.72.04
Operating System & Version	W10
Build System	VS2013

Is it possible to use depth aligned to color information to align infrared with color?

Thanks for your time!

[ddiakopoulos]

You can use one of the synthetic streams for this (color_aligned_to_depth). The intrinsics of the depth stream match the left IR sensor on the R200, and the single IR sensor on the F200. A pixel in this synthetic stream would match the corresponding IR pixel.

[tensorfreitas]

Thank you for your response! But the images acquired with color_aligned_to_depth are kind of a textured pointcloud. It is not possible to get an rgb image with the same coordinates as depth?

[ddiakopoulos]

@Goldesel23 I'm not quite sure I understand what you're trying to do. The images are generated from physically different locations so there will be missing data. I think when you say "textured pointcloud" you're referring to this new synthetic image with some missing data.

Take a look at the definitions of some of the synthetic streams the API supports here: https://github.com/IntelRealSense/librealsense/blob/master/include/librealsense/rs.hpp#L16

[tensorfreitas]

I'm trying to detect some keypoints in IR and Depth. And i would want to use them in Color images. I understand the images are different, but I am asking if it possible to use the conversion between depth to color for example to "convert" the location of the keypoints from depth/ir to color...

[ddiakopoulos]

@Goldesel23 color_aligned_to_depth is exactly what you want. If you find a keypoint in IR (matching intrinsics of the depth stream), then you can get the location of that keypoint in color via the synthetic stream. Keep in mind that not all pixels can be correctly projected, so you will be missing data.

This is all assuming F200. Keypoint detection for R200 in the IR image is ill-advised because of the texture being projected into the scene.... it's impossible to use for feature detection.

[tensorfreitas]

Thanks for the answer ddiakopoulos. I guess that will need to work for me!

Once again thanks for all the help!

[leslysandra]

hey @ddiakopoulos I am trying to use a "syntetic stream ": depth_aligned_to_color in LIBREALSENSE with UBUNTU, any hint how to use it?

[colbyham]

This way of handling projection makes temporal depth based foreground segmentation harder than it needs to be. :(

[mgserafi]

@ddiakopoulos, you mentioned

Keypoint detection for R200 in the IR image is ill-advised because of the texture being projected into the scene.... it's impossible to use for feature detection.

Could you elaborate on what this 'texture' is and if it's possible to remove? I want to perform alignment and I'm using the R200.

[kazoo-kmt]

@ddiakopoulos Do you know how the alignment process pipeline look like? For example, the depth aligned to color stream,

1. Get the depth image in the depth image coordinates
2. Deproject into the depth cloud coordinates by using the depth/IR camera's intrinsics
3. Transform the point to the color cloud coordinates by using the extrinsics from the depth to the color
4. Project into the color image coordinates Is the above understanding correct?