Failed to reproduce the quantitative results

minghanz commented 4 years ago

Hi RonaldSun,

Thanks for the open source code. I can now successfully run your code on the Euroc data sequences. Then I evaluated the accuracy of the trajectories using the tool by https://github.com/MichaelGrupp/evo. I tried both SE3 alignment and Sim3 alignment, but the reported RMSE is not the same as listed in the readme. I ran several times, and the rmse of V1_01 sequence is about 0.2 in both monocular and stereo mode, while the value in readme is 0.0599. Do you have any idea what might be the reason?

The script I used to run the program in stereo mode is as below, if it's helps.

dataname="V1_01_easy"
    # run dso
     ./build/bin/dso_dataset \
    files0=/media/minghanz/Seagate_Backup_Plus_Drive2/EuRoC_MAV_Dataset/${dataname}/mav0/cam0/data \
    calib0=/home/minghanz/VI-Stereo-DSO_original/VI-Stereo-DSO/calib/euroc/cam0.txt \
    imu_info=/home/minghanz/VI-Stereo-DSO_original/VI-Stereo-DSO/calib/euroc/IMU_info.txt \
    groundtruth=/media/minghanz/Seagate_Backup_Plus_Drive2/EuRoC_MAV_Dataset/${dataname}/mav0/state_groundtruth_estimate0/data.csv \
    imudata=/media/minghanz/Seagate_Backup_Plus_Drive2/EuRoC_MAV_Dataset/${dataname}/mav0/imu0/data.csv \
    pic_timestamp=/media/minghanz/Seagate_Backup_Plus_Drive2/EuRoC_MAV_Dataset/${dataname}/mav0/cam0/data.csv \
    preset=0 mode=1 \
    quiet=1 nomt=1 \
        savefile_tail=nt_${dataname}\
    imu_weight=6 imu_weight_tracker=0.6 stereo_weight=0.5 \
    use_stereo=1\
    files1=/media/minghanz/Seagate_Backup_Plus_Drive2/EuRoC_MAV_Dataset/${dataname}/mav0/cam1/data \
    calib1=/home/minghanz/VI-Stereo-DSO_original/VI-Stereo-DSO/calib/euroc/cam1.txt \
    pic_timestamp1=/media/minghanz/Seagate_Backup_Plus_Drive2/EuRoC_MAV_Dataset/${dataname}/mav0/cam1/data.csv \
    T_stereo=/home/minghanz/VI-Stereo-DSO_original/VI-Stereo-DSO/calib/euroc/T_C0C1.txt

The script I used to run evaluation in SE3 alignment mode is:

evo_ape euroc /media/minghanz/Seagate_Backup_Plus_Drive2/EuRoC_MAV_Dataset/V1_01_easy/mav0/state_groundtruth_estimate0/data.csv \
/home/minghanz/VI-Stereo-DSO_original/VI-Stereo-DSO/data/nt_V1_01_easy_ori_stereo.txt --align

Thanks for your help!

RonaldSun commented 4 years ago

@minghanz In my origin code, savetrajectory_tum() is used to save traj data. Use evo_ape tum ... for evaluation.

minghanz commented 4 years ago

@RonaldSun However, as in said in https://github.com/MichaelGrupp/evo/wiki/Formats:

In euroc mode, evo's metrics require you to give the reference in the EuRoC format and the estimated trajectory in the TUM format.

which is the case here. The ground truth file of Euroc dataset is in csv format so that tum mode cannot be used. So I think using euroc mode is correct here. Am I missing anything?

RonaldSun commented 4 years ago

@minghanz You can check https://github.com/RonaldSun/VI-Stereo-DSO/blob/master/src/FullSystem/FullSystem.cpp#L1206 I saved both dso and groundtruth trajs in tum format, so you can use the saved files in tum mode.

minghanz commented 4 years ago

@RonaldSun I freshly cloned and built this repo again. Then I run the V1_01 sequence. I run the command evo_ape tum ~/Repos/VI-Stereo-DSO/data/nt_V1_01_easy_gt.txt ~/Repos/VI-Stereo-DSO/data/nt_V1_01_easy.txt --align. or with -s. In monocular mode, the rmse is 0.18 with SE3 alignment, and 0.06 with Sim3 alignment. In stereo mode, the rmse is 0.052 with SE3 alignment, and 0.047 with Sim3 alignment. Are these results roughly aligned with your experiments? I did not see you specifying the mode (mono or stereo) in reporting the quantitative result. I assume it is stereo.

RonaldSun commented 4 years ago

My results are based on stereo mode. I haven't tested monocular results. Your results seem right.

RonaldSun / VI-Stereo-DSO

Failed to reproduce the quantitative results #25