Support for Stereo Images & Depth

[darshanAtSwaayatt]

Can I use this code to generate 3D reconstruction using Depth images obtained using Stereo images?

[Nik-V9]

Hi, yes, it is possible to extend our current RGB-D implementation to work with a Stereo setup. Support for two images as input will require significant restructuring of the code. Hence, this is currently not on our radar.

[darshanAtSwaayatt]

Hi @Nik-V9 ,

I have a stereo data and used "https://wiki.ros.org/stereo_image_proc" ROS package to generate depth images. I gave this data as input to the Splatam code. I updated the code with necessary scripts like the data.yaml, config script, dataset class, etc. but the output received is not proper. For the below scene (data is captured using the stereo setup mounted on a car. So the camera motion is in the forward direction only)

Corresponding depth image

the output generated was FRONT VIEW

SIDE VIEW

TOP VIEW

But using Iphone 12 pro data, the output was proper.

The config files updated by me for above data are in below zip: Splatam_updated_files.zip

Also wanted to understand how I can find "png_depth_scale" parameter for my data? As far as my understanding I can use below formula:

where

• f is the focal length
• B is the baseline distance between the stereo cameras,
• D_image : is the pixel value in the depth image.
• Z_meters : Real world depth in meters.

Z_meters will be the png_depth_scale for my data. But this will not be constant, as Z_meters is inversely proportional to D_image.

Can you tell what I am doing wrong? Any suggestions to fix my problem.

Thanks and Regards, @darshanAtSwaayatt

[darshanAtSwaayatt]

Hi @Nik-V9 ,

I have a stereo data and used "https://wiki.ros.org/stereo_image_proc" ROS package to generate depth images. I gave this data as input to the Splatam code. I updated the code with necessary scripts like the data.yaml, config script, dataset class, etc. but the output received is not proper. For the below scene (data is captured using the stereo setup mounted on a car. So the camera motion is in the forward direction only)

Corresponding depth image

the output generated was FRONT VIEW

SIDE VIEW

TOP VIEW

But using Iphone 12 pro data, the output was proper.

The config files updated by me for above data are in below zip: Splatam_updated_files.zip

Also wanted to understand how I can find "png_depth_scale" parameter for my data? As far as my understanding I can use below formula:

where

• f is the focal length
• B is the baseline distance between the stereo cameras,
• D_image : is the pixel value in the depth image.
• Z_meters : Real world depth in meters.

Z_meters will be the png_depth_scale for my data. But this will not be constant, as Z_meters is inversely proportional to D_image.

Can you tell what I am doing wrong? Any suggestions to fix my problem.

Thanks and Regards, @darshanAtSwaayatt

Hi @Nik-V9 , Can you please help.

[Nik-V9]

Hi, if the depth image you are saving has the pixel values in meters, the png_depth_scale will be 1.

Essentially png_depth_scale is a parameter to convert pixel values of a depth image to metric depth. In your case, after computing the metric depth if you are saving the image as is, the png_depth_scale will be 1.

Also based on the stereo depth image shared by you, there seems to be a lot of holes in the depth map. Without good multi-view coverage, I'm unsure how well the current version of SplaTAM will perform on the data.