search

weixk2015 / DeepSFM

This is a PyTorch implementation of the ECCV2020 paper "DeepSFM: Structure From Motion Via Deep Bundle Adjustment".
BSD 3-Clause "New" or "Revised" License
318 stars 45 forks source link

lower evaluation metrics using your trained models in google drive? #7

Closed zwxu064 closed 3 years ago

zwxu064 commented 3 years ago

Hi, I used your trained model, ground truth pose and depth files to calculate the "abs rel", "abs diff", .... But they are much lower than DPSNet (the last table of DPSNet code, https://github.com/sunghoonim/DPSNet).

Metrics 'abs_rel', 'abs_diff', 'sq_rel', 'rms', 'log_rms', 'a1', 'a2', 'a3'

MVS DPSNet | 0.0722 0.2095 0.0798 0.4928 0.1527 0.8930 0.9502 0.9760 DeepSFM | 0.0986,0.7810,0.2797,2.1187,0.1470,0.8889,0.9774,0.9916

Sun3D DPSNet | 0.1470 0.3234 0.1071 0.4269 0.1906 0.7892 0.9317 0.9672 DeepSFM | 0.0723,0.9464,0.2570,1.4316,0.1049,0.9598,0.9887,0.9942

RGBD DPSNet | 0.1538 0.5235 0.2149 0.7226 0.2263 0.7842 0.8959 0.9402 DeepSFM | 0.1353,3.8738,2.2566,18.1695,0.2085,0.8909,0.9722,0.9836

Scenes DPSNet | 0.0558 0.2430 0.1435 0.7136 0.1396 0.9502 0.9726 0.9804 DeepSFM | 0.0668,1.0298,0.2532,1.9122,0.1057,0.9621,0.9906,0.9961

I used ground truth poses.txt and *.npy depth files, which should give a better performance. But from the above comparison, I found in the first 5 metrics (lower better), at most cases, DeepSFM is worse than DPS, especially DeepSFM on RGBD. This seems a bit wired. Is there any issues about this problem or could you give me some tips as I may miss something?

weixk2015 commented 3 years ago

Thanks, our depth-subnet pretrained model is trained with estimated pose and depth initializations, since the task is to estimate depth and pose without gt camera motion, and the results are not consistently better than DPSNet which trained with GT pose. Meanwhile, in my experiments, I found that the Abs diff Sq_Rel and RMSE are very sensitive metrics.

zwxu064 commented 3 years ago

got it, thank you.

wangxiyuan9 commented 2 years ago

@zwxu064 @weixk2015 hello , I think you have reproduced the paper .It seems because the initial posture data from demon is quite different from pose.txt , so my evaluation results are very different from the paper. After inspection , I don't know why and how to do? I get these initial poses which are obtained by running the example.py file in the demon with given two photos. Could you share how you obtained the initial poses from the demon and the level of difference between them and pose.txt? thanks very much