search

ethz-asl / maplab

A Modular and Multi-Modal Mapping Framework
https://maplab.asl.ethz.ch
Apache License 2.0
2.57k stars 722 forks source link

RMSE Error on EuRoC Dataset #260

Open YilmazKadir opened 4 years ago

YilmazKadir commented 4 years ago

Hello, I am evaluating ROVIO algorithm in VIO mode. In the paper it is stated that in EuRoC MH_01 data they achieve 0.178m RMSE, and in V2_01 they achieve 0.064 m RMSE. Which alignment procedure are you using? I am using evo package and I achieve 0.213885 m RMSE for MH_01 and 0.103967 m RMSE for V2_01. Can you explain why I can not achieve the same results?

How I calculate RMSE: I run the algorithm in VIO mode and save the map. I open the maplab console, load the map and export the vertices using csv_export. I convert the vertices.csv file into TUM format. I use evo_ape euroc command with alignment (-a) and these are the results I get for MH_01:

MH01_ROVIO MH01_ROVIO_traj

APE w.r.t. translation part (m) (with SE(3) Umeyama alignment)

   max  0.987680
  mean  0.178013
median 0.162161
   min  0.006351
  rmse  0.213885
   sse  83.213396
   std  0.118567
amichayfeldman commented 4 years ago

I also got strange behavior. I didn't get the paper's results and in addition I got with ROVIOLI worse results than VIO mode in MH1 data.

araujokth commented 4 years ago

Have you tried running maplab from the pre-release-march-2018 branch https://github.com/ethz-asl/maplab/tree/pre_release_public/march-2018? As they state in the release notes you may be able to achieve better results https://github.com/ethz-asl/maplab/pull/55

Please see my updated comment on https://github.com/ethz-asl/maplab/issues/261#issuecomment-675296188, I think this solves your problem