MARSLab-UMN / MARS-VINS

MARS-VINS binary release repository
http://mars.cs.umn.edu/
Other
116 stars 41 forks source link

EuRoC Evaluation #8

Closed wtrobin closed 6 years ago

wtrobin commented 6 years ago

Hi,

I've tried to evaluate your tracker's performance on the (stereo) EuRoC data set. So far I've converted all the images (.png -> .pgm), converted the IMU data file to comply to the MARS-VINS format, and I've adapted the sample_config.txt to the EuRoC camera calibration and IMU values.

After a few tiny tweaks in the generic_driver.cpp (to be able to read EuRoC images instead of m0000012.pgm) I'm finally able to start the example application. However, it crashes somewhere inside the binary and I'm not sure what I can do about that.

The attached Valgrind output highlights two mismatched new/delete pairs (during image loading). The crash is caused by an invalid read in Estimators::StereoInitializerEngine::set_imu_measurement()

Any help would be appreciated.

Thanks, Werner

P.S. If fixing this is tricky, I'd also be happy with absolute trajectory errors (RMSE) for all individual EuRoC sequences :-)

paulmrinal commented 6 years ago

Hi,

Sorry for your trouble. The drivers on this branch were experimental and might have some bugs. I will have a look into it. However, I am currently a bit pre-occupied and it might take some time before I am able to fix the bug.

In the mean time, if you are ok with having the absolute RMSE data, I am listing them here.

Dataset Absolute RMSE (m)
MH_01_easy 0.17501
MH_02_easy 0.063584
MH_03_medium 0.098886
MH_04_difficult 0.3326
MH_05_difficult 0.309
V1_01_easy 0.066619
V1_02_medium 0.055443
V1_03_difficult 0.09485
V2_01_easy 0.047206
V2_02_medium 0.071117

The results are using this configuration: max_number_of_extracted_keypoints = 400, sliding_window_size = 5, image_processing_rate = 30 Hz (to match the default configuration of OKVIS). The released binaries, however, have different configuration (max_number_of_extracted_keypoints = 600, sliding_window_size = 10, image_processing_rate = adaptive, parameters tuned for wide FOV fisheye cameras), so the results will vary.

P.S. The V2_03_difficult dataset seemed to have some issues (do not remember what exactly was the problem), so I did not include it in my results.

wtrobin commented 6 years ago

Hi,

thanks a lot for your very swift reply! I think the problem is on my side; I've specified the IMU rate incorrectly... nevermind :-)