search

cleinc / bts

From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
GNU General Public License v3.0
635 stars 179 forks source link

Evaluation results in the paper #52

Closed tjqansthd closed 4 years ago

tjqansthd commented 4 years ago

Hi, thanks for your great work!

I also use both raw velodyne scans and official ground truth on KITTI in my work, so I am referring to your paper. In the paper, the results in other methods have slightly different values presented in their paper. (i.e., DORN a1 0.932 a2 0.984 a3 0.994 rel 0.072 sq rel 0.307 rmse 2.727 rmse log 0.120 in their paper, and a1 0.936 a2 0.986 a3 0.995 rel 0.081 sq rel 0.337 rmse 2.930 rmse log 0.121 in your paper as well as results of LRC. )

Could you explain how did you get the quantitative results of other work in your paper?

cogaplex-bts commented 4 years ago

Originally we ran the authors' code by ourselves and got the results. However, we understand that there can be some mismatch in the verification process between ours and theirs. So we will soon update our paper in arxiv, and that version will contain the same values with the paper from Fu et al. (DORN).

tjqansthd commented 4 years ago

Thank you for your reply! Is the LRC 2017 result of official g.t also obtained by your experiment?

jahaniam commented 4 years ago

@tjqansthd Please use official gt for evaluation (at least provide this number in your papers and say it is based on raw lidar or official g.t). Based on my experiments raw lidar has too much error in KITTI.

@tjqansthd below is the evaluation of LRC and raw lidar using official g.t. I've done. image

Also whoever works on depth estimation on KITTI please see https://github.com/mrharicot/monodepth/issues/166#issue-334733737 for more clarification.

cogaplex-bts commented 4 years ago

@tjqansthd Yes, it is also from our experiments with the authors' code (monodepth).

tjqansthd commented 4 years ago

@jahaniam Thank you! The results give me a great help. I know the official g.t is more desirable but many previous works have the evaluation results in raw LiDAR more than official g.t yet.

@cogaplex-bts

Now the confusion was solved. Thank you very much!