How are Euroc ground truths obtained in ov_data

[HieuPhan33]

Hi, thanks for open-sourcing a useful VIO tool.

I'm evaluating the estimated trajectory of Euroc V1_01_easy. I saw a ground truth file V1_01_easy.csv/txt files in ov_data/euroc_mav. When I checked with the ground truth file in the Euroc souce: http://robotics.ethz.ch/~asl-datasets/ijrr_euroc_mav_dataset/vicon_room1/V1_01_easy/V1_01_easy.bag. The ground truth is stored in topic: /vicon/firefly_sbx/firefly_sbx. I can see that the source ground truth and open_vins ground truth are different.

When I evaluate the estimated trajectory path_est.txt with the open_vins ground truth, the performance is solid. However, the trajectory drifts away from the source ground truth.

Output for evaluating on open_vins gt:

[COMP]: 2895 poses in V1_01_easy => length of 58.35 meters
[TRAJ]: q_ESTtoGT = -0.006, -0.004, -0.143, 0.990 | p_ESTinGT = 0.866, 2.179, 0.940 | s = 1.00
[COMP]: 2680 poses in 1595400591_estimate => length of 57.11 meters

Output for evaluating on source gt:

[COMP]: 14629 poses in V1_01_gt => length of 60.12 meters
[TRAJ]: q_ESTtoGT = 0.076, -0.535, 0.059, 0.839 | p_ESTinGT = 0.770, 2.178, 1.073 | s = 1.00
[COMP]: 2680 poses in 1595400591_estimate => length of 57.11 meters

[goldbattle]

I believe that topic is of the vicon marker MAV frame, which does not coincide with the IMU frame that most estimation methods estimate. OpenVINS publishes the IMU sensor frame.

The csv files for groundtruth are pulled from the ASL zip files (should be a groundtruth folder in it). This is then converted using the ov_eval format_converter utility (this is the text file in the repo).

I am surprised with the large error you have shown in the first figure, what method are you using to align? Typically we use posyaw to align it to find the ATE / RMSE of the trajectory. This corresponds to the 4 unobservable DOF that VINS has and are arbitrarily chosen on startup.

On Wed, Jul 22, 2020 at 10:17 PM StevePhan101 notifications@github.com wrote:

Hi, thanks for open-sourcing a useful VIO tool.

I'm evaluating the estimated trajectory of Euroc V1_01_easy. I saw a ground truth file V1_01_easy.csv/txt files in ov_data/euroc_mav. When I checked with the ground truth file in the Euroc souce: http://robotics.ethz.ch/~asl-datasets/ijrr_euroc_mav_dataset/vicon_room1/V1_01_easy/V1_01_easy.bag. The ground truth is stored in topic: /vicon/firefly_sbx/firefly_sbx. I can see that the source ground truth and open_vins ground truth are different.

When I evaluate the estimated trajectory path_est.txt with the open_vins ground truth, the performance is solid. However, the trajectory drifts away from the source ground truth.

Output for evaluating on open_vins gt:

[COMP]: 2895 poses in V1_01_easy => length of 58.35 meters [TRAJ]: q_ESTtoGT = -0.006, -0.004, -0.143, 0.990 | p_ESTinGT = 0.866, 2.179, 0.940 | s = 1.00 [COMP]: 2680 poses in 1595400591_estimate => length of 57.11 meters

[image: eval_ov] https://user-images.githubusercontent.com/32691090/88246417-c731f480-ccdd-11ea-867f-a898d801d474.PNG [image: image] https://user-images.githubusercontent.com/32691090/88246506-0fe9ad80-ccde-11ea-8243-cbe039b218f3.png

Output for evaluating on source gt:

[COMP]: 14629 poses in V1_01_gt => length of 60.12 meters [TRAJ]: q_ESTtoGT = 0.076, -0.535, 0.059, 0.839 | p_ESTinGT = 0.770, 2.178, 1.073 | s = 1.00 [COMP]: 2680 poses in 1595400591_estimate => length of 57.11 meters

[image: eval_src] https://user-images.githubusercontent.com/32691090/88246426-d153f300-ccdd-11ea-8368-28b9431e3c16.PNG [image: image] https://user-images.githubusercontent.com/32691090/88246537-2bed4f00-ccde-11ea-977b-2d808188ec50.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/rpng/open_vins/issues/88, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQ6TYUZQ5YH6E63IPOPQKLR46MUJANCNFSM4PFI736A .

[HieuPhan33]

Thanks for the quick response.

I was using se3single method for evaluation. I have converted the data.csv from the source to txt using format_converter.cpp.

Yes, you are right, using posyaw yields better result. Here is the output when using posyaw evaluation,

Evaluation on source_gt.txt:

[COMP]: 28712 poses in v1_gt => length of 58.59 meters
[TRAJ]: q_ESTtoGT = 0.000, 0.000, -0.139, 0.990 | p_ESTinGT = 1.408, 2.136, 0.996 | s = 1.00
[COMP]: 2678 poses in 1595400591_estimate => length of 57.11 meters

Evaluation on openvins_gt.txt:

[COMP]: 2895 poses in V1_01_easy => length of 58.35 meters
[TRAJ]: q_ESTtoGT = 0.000, 0.000, -0.137, 0.991 | p_ESTinGT = 1.427, 2.151, 0.999 | s = 1.00
[COMP]: 2680 poses in 1595400591_estimate => length of 57.11 meters

I can notice that the ov_gt.txt and src_gt.txt are slightly different in terms of both timestamp and pose values. Even though the difference is insignificant, I'm still curious why the difference occurs. Additionally, I also see the files V1_01_easy_original and V1_01_easy and they are different as well. Could you please give me some clarifications?

Thank you in advance.

[goldbattle]

That difference should be described here on the dataset page: https://docs.openvins.com/gs-datasets.html#gs-data-euroc  

Groundtruth on V1_01_easy We have found that the groundtruth on the V1_01_easy dataset is not accurate in its orientation estimate. We have recomputed this by optimizing the inertial and vicon readings in a graph to get the trajectory of the imu. You can find the output at this link and is what we normally use to evaluate the error on this dataset.

[HieuPhan33]

Thanks for your information.

[goldbattle]

For those that re-visit this issue, please see the vicon2gt repository if you want to generate this yourself: https://github.com/rpng/vicon2gt

[habib-Boloorchi]

Hi Do you know if the data in ground truth is the distance from origin or the distance from last timestamp.