search

castacks / DytanVO

[ICRA'23] DytanVO: Visual Odometry in Dynamic Environments
BSD 3-Clause "New" or "Revised" License
173 stars 20 forks source link

Get inconsistent results with paper #20

Closed Xie-HIT closed 8 months ago

Xie-HIT commented 8 months ago

I follow the instructions and successfully run DytanVO. But the ATE seems to be inconsistent with your paper (Table II). For sequence 07_1, I got ATE=1.6556. However, Table II in the paper gives ATE=0.6367. I thought maybe the results should convert from NED coordinate to camera coordinate, so I slightly modified the code, but again got inconsistent results with ATE=1.5879 for 07_1. Inconsistent results also occur in 01_1. Then I found an helpful issue here: #18 , where the author said they used 136 frames of 07_1. However, I download DynaKITTI followed your instruction, find the 07_1 has 475 frames. Could you share which 136 frames you use ? The frame number of 04_0 and 08_0 also mismatch (04_0: 271 vs 36, 08_0: 46 vs 40). In addition, the pretrained weight provided by the instruction does not have vonet_ft.pkl (only have vonet.pkl). Maybe I think is reason of the inconsistent problem.

Xie-HIT commented 8 months ago

There are some randomness on cow mask. This maybe an another reason.

SecureSheII commented 8 months ago

Thanks for your comment! The mismatch of the frames must be the cause. Out of the 475 frames you've downloaded for 07_1, we must have trimmed the clip to be 136 frames that just include dynamic objects (I believe the remaining ~300 frames are mostly without any dynamic object). Same happens to 04_0. Currently I might not have enough capacity to retrieve the exact 136 frames for you but you can simply play the sequence and trim it on your own, along with the pose file. We should've prepared for the data more carefully before release. Thanks for the catch though!

Xie-HIT commented 8 months ago

Thanks for your quick reply! I clip the first 136 frames of 07_1 (000626.png - 000761.png, assume the author used the first 136 frames). Considering the randomness in cow mask, I average five runs. And reproduce ATE=0.7441. In addition, I evaluate the same result using evo tools, and get ATE=0.5602. By average the results of different evaluation tools, ATE=0.6522. This value is close to the paper, I believe it is within the acceptable range. Thanks for your reply again, it's helpful !