search

facebookresearch / DistDepth

Repository for "Toward Practical Monocular Indoor Depth Estimation" (CVPR 2022)
Other
216 stars 20 forks source link

Question about scaling factor in evaluation script #15

Open afazel opened 1 year ago

afazel commented 1 year ago

I was wondering what is the motivation for using this scaling factor during evaluation? depth_pred *= torch.median(depth_gt) / [torch.median(depth_pred)](url) https://github.com/facebookresearch/DistDepth/blob/dde30a4cecd5457f3c4fa7ab7bf7d5c8ea95934a/execute_func.py

If we remove this scaling, the results are not aligned with what is reported in the paper. For NYUv2 dataset, the RMSE increases from 0.58 to 0.99

Can you please explain why did you use that scaling during evaluation? Is it reasonable to scale-up from ground truth information?

Also in [compute_depth_errors()](https://github.com/facebookresearch/DistDepth/blob/0414b6a30f3fc7310d29d4ccb8062d4a597ca3a2/layers.py#:~:text=a1%20%3D%20(thresh%20%3C%201.25,3).float().mean()), how did you decide on 1.25 as a threshold?

choyingw commented 1 year ago

We follow the previous protocols for self-supervised method to evaluate on NYUv2, where median scaling is applied. See P2Net and StructDepth.

1.25 is a conventional usage in the field to measure depth accuracy. See https://arxiv.org/pdf/2003.06620.pdf