Depth estimation techniques

[martalopezbrea]

Useful codes for depth estimation:

• https://github.com/lppllppl920/EndoscopyDepthEstimation-Pytorch

• Extremely Dense Point Correspondences using a Learned Feature Descriptor https://github.com/lppllppl920/DenseDescriptorLearning-Pytorch This code is the same as the previous one but uses COLMAP as SfM method.

• High Quality Monocular Depth Estimation via Transfer Learning https://github.com/ialhashim/DenseDepth

[mabel-lr]

ToDo: generate depth maps with same pixel value range as the depth maps that STTR outputs. idea: open their depth maps in python and look at the pixel value range, then look for techniques that output a depth map in the same pixel value range

their technique is not valid for us because their source is a stereo video and this means that it provides right and left images, we don't have this as our videos are not stereoscopic. we have to find the depth map from a single 2d image

[martalopezbrea]

Their depth maps are grayscale images. In an image with grayscale, the pixel values typically range from 0 to 255. I've already tried it in python and the maximum result of one of them is 0-253.

[mabel-lr]

I got this code running: http://stereo.jpn.org/jpn/stphmkr/google/colabe.html#omake here the results (it is similar for all the frames):

unfortunately, it is a model trained on persons, and it does not offer good results for our project.

maybe we can take it as a base to perform modifications if we do not find anything better, as we know it runs and provides an output.

[martalopezbrea]

MiDaS technique: https://github.com/isl-org/MiDaS I used the "dpt_beit_large_512" model

The result is really good I think.

[martalopezbrea]

I also tried it with this website: https://huggingface.co/spaces/radames/dpt-depth-estimation-3d-obj The prediction depth is not really good:

But the 3d reconstruction is great:

Try it yourselves to appreciate the reconstruction

[martalopezbrea]

NOT NECCESSARY (info on next card):

Code to convert RGB image to grayscale:

https://colab.research.google.com/drive/13f86MZr_vV7M41yiFdDCFxhQWl4GYn_G?usp=drive_link

import cv2 import numpy as np

image = cv2.imread('0027-dpt_beit_large_512.png')

if(len(image.shape)<3): print('gray') elif len(image.shape)==3: print('Color(RGB)')

Convert the RGB image to grayscale gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) Convert grayscale image to array array_image = np.array(gray_image)

Show image import matplotlib.pyplot as plt plt.imshow(array_image, cmap='gray') plt.axis('off') plt.show()

[mabel-lr]

so that the output is grayscale just add --grayscale to the line in the cmd. the right line is:

python run.py --model_type dpt_beit_large_512 --input_path input --output_path output --grayscale