search

thuzhaowang / idn-solver

ICCV 2021 code release of paper "A Confidence-based Iterative Solver of Depths and Surface Normals for Deep Multi-view Stereo"
MIT License
50 stars 3 forks source link

ScanNet evaluation doesn't match paper #3

Open mrharicot opened 2 years ago

mrharicot commented 2 years ago

Hi, Thank you for sharing your code! I ran test on scannet following the instructions but I get much worse results. Do you know what might be happening?

Total test num: 10434
['abs_rel', 'abs_diff', 'sq_rel', 'rms', 'log_rms', 'a1', 'a2', 'a3']
[0.26725563624661075, 0.4768538641935877, 0.18805316813831266, 0.5750360812767931, 0.3188030923313161, 0.4764993420147505, 0.8155114483545208, 0.9528344624353279]

Here's the first output for scene0707_00 image

thuzhaowang commented 2 years ago

Hi,

Thanks for your attention!

I checked the code and it runs well here for me. May I ask, are your testing environments the same as the those in environment.yml? We used an old Pytorch version (1.1.0) since we built the code based on NAS. It might have some inconsistency with the latest version.

mrharicot commented 2 years ago

I am using pythorch 1.7.1 as my GPU only supports CUDA 11.0 and up. However I have used newer versions of pytorch on older ones and never had issues. Are the pretrained weights on google drive the correct ones?

thuzhaowang commented 2 years ago

I see. I have tried to use Pytorch 1.7.1 right now and it indeed produces bad results, while the original environment works well. I will investigate where the inconsistency is, in these days. Before that, probably you could find another environment using the same environment (Pytorch 1.1.0) to run the evaluation.

mrharicot commented 2 years ago

Thanks for your help! I see that someone seems to have a similar issue for NAS on pytorch 1.3.1 so it's possible the backwards compatibility breaking change happened between 1.1. and 1.3.

thuzhaowang commented 2 years ago

Hi, I just found the change of torch.nn.functional.grid_sample is the problem. It affects the construction of feature cost volume.

"Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details."

So change the the grid sample in https://github.com/thuzhaowang/idn-solver/blob/6b5dcfd94f35cc118c5dee0f98401e4848e670e3/core/utils/inverse_warp.py#L108 and https://github.com/thuzhaowang/idn-solver/blob/6b5dcfd94f35cc118c5dee0f98401e4848e670e3/core/utils/inverse_warp_d.py#L123 with align_corners=True will solve the problem for higher version of Pytorch (1.3.0 or later).

mrharicot commented 2 years ago

This looks a lot better, thanks! image