Reconstruct images from Structure sensor

[mintos002]

Hi there, I would like to know if there is a way to reconstruct from some RGBD images with some Depth only images in C++. The issue is that the Structure sensor gets constantly depth images and only few colored images, so is there any way to combine RGBD images with Depth only images? or is there any example of reconstruction with depth only images?

Thanks :)

[syncle]

Hi @mintos002. This issue is duplication of #482.

The idea is to put dummy color image to the pipeline. Try this by modifying this function. For example,

def read_rgbd_image(color_file, depth_file, convert_rgb_to_intensity, config):
    color = read_image(color_file)
    depth = read_image(depth_file)
    if config["depth_map_type"] == "redwood":
        rgbd_image = create_rgbd_image_from_color_and_depth(color, depth,
                depth_trunc = config["max_depth"],
                convert_rgb_to_intensity = convert_rgb_to_intensity)
    return rgbd_image

This would be modified like the below script:

def read_rgbd_image(color_file, depth_file, convert_rgb_to_intensity, config):
    color = read_image("your_dummy_image_having_same_resolution_as_depth.jpg")
    depth = read_image(depth_file)
    if config["depth_map_type"] == "redwood":
        rgbd_image = create_rgbd_image_from_color_and_depth(color, depth,
                depth_trunc = config["max_depth"],
                convert_rgb_to_intensity = convert_rgb_to_intensity)
    return rgbd_image

Edited: @mintos002: I just saw you have requested c++ example for this. :) Please regard my comment as a quick proposal.

[qianyizh]

Regarding Structure sensor, I think you can actually stream both depth and color images (put in a video file) at ~30fps. I did it before with a very old version of Structure sensor SDK. I think you can still do it. Having said that, I am not a fan of the iPhone color camera: it is distorted quite a bit - so not a good input for odometry.

The workaround, as @syncle mentioned, is to disable color stream completely and rely on a frame-to-frame depth image registration.

However, I do think Open3D needs a frame-to-model registration scheme like KinectFusion, which has not been implemented yet.

[mintos002]

Ok. I'll try to search the old Structure SDK, if I don't find it I'll use dummy images, or try to use a RealSense camera!

Thank you all so much!

[www158]

Hi @mintos002. This issue is duplication of #482.

The idea is to put dummy color image to the pipeline. Try this by modifying this function. For example,

def read_rgbd_image(color_file, depth_file, convert_rgb_to_intensity, config):
    color = read_image(color_file)
    depth = read_image(depth_file)
    if config["depth_map_type"] == "redwood":
        rgbd_image = create_rgbd_image_from_color_and_depth(color, depth,
                depth_trunc = config["max_depth"],
                convert_rgb_to_intensity = convert_rgb_to_intensity)
    return rgbd_image

This would be modified like the below script:

def read_rgbd_image(color_file, depth_file, convert_rgb_to_intensity, config):
    color = read_image("your_dummy_image_having_same_resolution_as_depth.jpg")
    depth = read_image(depth_file)
    if config["depth_map_type"] == "redwood":
        rgbd_image = create_rgbd_image_from_color_and_depth(color, depth,
                depth_trunc = config["max_depth"],
                convert_rgb_to_intensity = convert_rgb_to_intensity)
    return rgbd_image

Edited: @mintos002: I just saw you have requested c++ example for this. :) Please regard my comment as a quick proposal.

How did you construct the "dummy image", can you share the "dummy image"?