Evaluation metric and ground-truth source

[glennliu]

Hi Xu,

Thanks for your inspiring work. In Table V. of your paper, you reported the relative localization result of your experiment.

1. How do you calculate the relative localization error? Kimera-multi uses average trajectory error(ATE) to compute the localization error. Do you use the same metric?
2. How do you get the ground-truth transformation between robots? In computing the relative localization error, do you compute th error in all the keyframes, or it is just the covisbile keyframes between a pair of agent?

I cannot find the related illustration in the paper.

Thanks for your time Chuhao

[XuRobotics]

Hi Chuhao,

Thanks for your interest! Good questions.

1. Yes, currently we only perform inter-robot localization evaluation. We are working on some benchmark tests with public datasets and we are planning to add in metrics such as ATE, RPE, or drift rate soon. You can check our Arxiv paper for future updates.
2. We did this through two approaches: (1) aligning the robot's starting pose with a pre-set marked position (e.g. a landing pad for UAVs), and (2) getting ground-truth relative transformation between robots using manual measurements (point laser and tape measure). We also have GPS data for outdoor, which can potentially serve as additional ground truth along the trajectories. We compute the error using only the reference frame (i.e. the gravity-aligned starting pose of each robot) transformations between robots.

Best, Xu

[glennliu]

Hi Xu

Thanks for your time. If possible, I hope you can illustrate the calculation mathematically in your updated paper.

For example, if you use pre-set marked position, I assume there is only one keyframe have a ground-truth relative transformation between a pair of agent. If you use RTK, almost every pair of keyframe have a ground-truth relative transformation. Do you use all of them to compute an averaged error?

Regards Chuhao

[XuRobotics]

" if you use pre-set marked position, I assume there is only one keyframe have a ground-truth relative transformation between a pair of agent." -- Yes this is correct.

"If possible, I hope you can illustrate the calculation mathematically in your updated paper." -- Of course, we will try to add some explanation of this into the updated arXiv!

Thanks!

Best, Xu