search

JiatianWu / stereo-dso

Stereo DSO
GNU General Public License v3.0
211 stars 85 forks source link

KITTI error is higher than your reported numbers #5

Open fishcu opened 6 years ago

fishcu commented 6 years ago

Hi,

Thanks for making stereo DSO available.

I have evaluated KITTI benchmark (sequences 00 and 05) across 5 runs of stereo DSO (to account for randomness). The resulting errors are higher than what you report, especially the translational drift. You report the following: 2.5% trans. error, 0.0045 deg/m rot. error on KITTI 00 2.7% trans. error, 0.0035 deg/m rot. error on KITTI 05. I found the following: 3.6% trans. error, 0.0040 deg/m rot. error on KITTI 00 3.0% trans. error, 0.0043 deg/m rot. error on KITTI 05.

Could you please tell me if you have tuned any parameters to reach better KITTI results? For example, does your coupling factor correspond to the coupling factor of the original publication (which relates the static stereo matching energy to the frame-to-frame energy)?

I have attached my results of KITTI 00 and 05. The graphs that do not show the estimated trajectory correspond to the average across 5 runs. I have also found that stereo DSO crashes when it loses track.

KITTI_00_kitti_plots_run_avg.pdf KITTI_00_run0_kitti_plots.pdf KITTI_05_kitti_plots_run_avg.pdf KITTI_05_run0_kitti_plots.pdf

NikolausDemmel commented 6 years ago

What paper are you referring to?

fishcu commented 6 years ago

I am referring to the errors reported on the main page (README.md), under "4. Experiments". Since the authors report a range of errors, I have taken the average of the two values reported on the main page for comparison.

BTW, both of these results are worse than the ranking in the official KITTI benchmark of the non-public research implementation, so either the research implementation is somehow better, or it was better tuned for KITTI. I can't link there right now as the website has been down for several days now.

NikolausDemmel commented 6 years ago

Not sure about the errors reported here. Just make sure you don't mix up this repository and https://vision.in.tum.de/research/vslam/stereo-dso, which are independent works. Not sure about what is on the KITTI ranking page.

fishcu commented 6 years ago

Yes, I'm aware of that. However, AFAIK, you / the TUM people have not published source code, right? For that reason I evaluated this implementation here.

fishcu commented 6 years ago

Anyway, these questions remain:

  1. Is there a probable reason why my evaluation produces worse results than the ones published on this repository? For example, did the authors of this repository tune some parameters additionally without pushing them to the repo?
  2. What are the reasons for a discrepancy of the results reported in the arxiv paper and the ones of this repo? Translational drift is about 3x worse, rot. drift is about 2x worse (although sample size is unknown / very small).
NikolausDemmel commented 6 years ago

Yes, I'm aware of that. However, AFAIK, you / the TUM people have not published source code, right? For that reason I evaluated this implementation here.

Yes, the code is not available unfortunately.

What are the reasons for a discrepancy of the results reported in the arxiv paper and the ones of this repo? Translational drift is about 3x worse, rot. drift is about 2x worse (although sample size is unknown / very small).

I guess there can be many reasons, since it is a different system and a different implementation. I'm not sure if there is a paper describing the system in this repository, so it is hard to compare. I guess what you see on the KITTI benchmark website is Wang et al. as well and not this repo.

JiatianWu commented 6 years ago

Sorry for late reply. I use code KITTI provide to measure the output results. Their method is pretty complicated, maybe you should test the output with KITTI evaluation code.

JiatianWu commented 6 years ago

Also, I implemented stereo DSO by following how stereo LSD does because the official stereo DSO paper has not published when I did this. I always want to reimplement my code but did not find time. You are welcomed to modify my version following the stereo DSO paper.

fishcu commented 6 years ago

@JiatianWu Thank you for the explanations! Interesting to see that this implementation uses stereo-LSD's algorithm!

I have re-implemented the KITTI benchmarks with some python scripts. The plots you see are generated from my scripts. Their method is not that complicated, it's just not documented very well. :smile:

In any case, on the website I see only Wang et al.'s results. How did you test your algorithm with KITTI's evaluation code? From my understanding, you have to send them your results personally (and they will put it up in the rankings).

JiatianWu commented 6 years ago

You do not need to send code to get results. KITTI evaluation code is available on their website, just download and run it to get results..

fishcu commented 6 years ago

Thank you JiatianWu, that clears some confusion.

Did you run your implementation with the settings/parameters that are currently available in the repository?

2018-04-29 6:39 GMT+02:00 JiatianWu notifications@github.com:

You do not need to send code to get results. KITTI evaluation code is available on their website, just download and run it to get results..

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JiatianWu/stereo-dso/issues/5#issuecomment-385225206, or mute the thread https://github.com/notifications/unsubscribe-auth/AYLnmhjMh9PeLh8MU44RGwLb1QNij8rgks5ttUQDgaJpZM4TjOYk .

yunxiaoshi commented 5 years ago

Hi @fishcu

Can you share how you did the evaluation and results plotting? I'm new to this field and from my understanding I would need kitti_xx_pose_estimated.txt and kitti_xx_pose_gt.txt, i. e. how do I get the estimated pose from stereo dso and dump them into a text file?

A million thanks if you could shed some light on this!

Pedrous commented 5 years ago

Hey @fishcu,

How were your results with this stereo-dso? I am having a quite disturbing scale drift with the method. I guess it should be because of the baseline. It's just that at the same time with the same parameters I am getting very good results ORB-SLAM2 stereo version. That is why I doubt that my camera parameters could be so wrong. I am processing a dataset that I collected by myself.

How did you rectify the images? I used openCV and calculated the baseline from the projection matrix after rotating the images in to the same plane. But as I told, I get good results for ORB-SLAM2 with the same baseline, so I am curious if you noticed the same effect?