search

raulmur / ORB_SLAM2

Real-Time SLAM for Monocular, Stereo and RGB-D Cameras, with Loop Detection and Relocalization Capabilities
Other
9.3k stars 4.69k forks source link

Failed to replicate results in Kitti Dataset #753

Open FranRaverta opened 5 years ago

FranRaverta commented 5 years ago

I have a problem trying to replicate the stereo results shown on Mur-Artal's paper in the sequence 03 of the Kitti Dataset. While the paper shows an absolute error of 0.6m and a relative error of 0.71%, I could only get up to 4.8m and 3.06%, respectively.
I would also like to mention that if I use the yaml file that corresponds to the sequences 04 to 11, the results gotten are closer to the ones published (at about 1.19m and 1.99%). I would like to know if any of you had the same problem, and how did you solve it in that case.

I am currently running the algorithms with a i7-4790k and 16 GB of ram memory.

RunqiuBao commented 4 years ago

I also had this problem.
When using orb-slam stereo to run sequence 05 of kitti, the result ATE is around 3.2m rather than 0.8m in the paper.

Did you solve this issue? @FranRaverta

FranRaverta commented 4 years ago

Hello @RunqiuBao. I couldn't solve this problem still, so if you eventually come across a solution, I would be happy to hear it. By the way, are you using any align method to get the results? If I'm not mistaken, the results shown in the paper include this step. I got the above results using the method of Umeyama, but any of the alternatives should work.

RunqiuBao commented 4 years ago

Hi, @FranRaverta I did not use any alignment method. I suppose for kitti benchmark, we do not have to use alignment before evaluation. But, I did notice if you use alignment, you could make ATE much smaller. Especially if you also adjust the scale, ATE can be reduced from 3.2m to 0.4m for sequence 05. But still, I have no idea how did they get 0.8 ... Let's see if I could figure out.

sazak1 commented 4 years ago

Hello @RunqiuBao and @FranRaverta, I ran the ORB-SLAM code with EuRoC and KITTI using stereo vision. But I couldn't get overlapping trajectory and ground-truth. Probably I will have large errors similar to yours. Did you solve your problem? Actually I do not understand to adjust scale factor or anything else manually if we don't work with monocular.

FranRaverta commented 4 years ago

Hello @didnotwork, I'm not sure to understand your problem, but I think it might be because the estimated trayectory and the ground-truth may noy be in the same coordinate axes, and therefore cannot be properly compared. If that's the case, you could solve that with the same algorithms that are used to estimate the scale factor when working with monocular cameras (like Umeyama or Horn algorithms, for example).