MichaelGrupp / evo

Python package for the evaluation of odometry and SLAM
https://michaelgrupp.github.io/evo/
GNU General Public License v3.0
3.33k stars 745 forks source link

peculiar issue when I used Umeyama's method #624

Open TwiceMao opened 5 months ago

TwiceMao commented 5 months ago

@MichaelGrupp

Description: According to my understanding of Umeyama's method, it first aligns two sets of poses through similarity transformations and then I can calculate their distances. When using your Umeyama's method... (with scale correction), I encountered a peculiar issue. Simply put, the distances between the estimated poses and the ground truth poses, before and after rigid body transformation, are different. Specifically, All the camera poses I use are transformations from the camera to the world coordinate system,i.e. poses I used: camera to world(c2w). I suspect whether it is necessary to use 'world to camera' as the pose instead of 'camera to world'.I have a set of ground truth poses named GT_poses.txt and another set estimated through some method named estimated_poses.txt. Additionally, I normalized the first frame of GT_poses.txt. To be specific, I extracted the poses of the first frame from GT_poses.txt, and then for each frame, I multiplied its poses with those of the first frame to obtain GT_poses_setorigin.txt. I observed significant differences in distances, both in translation and rotation, between estimated_poses.txt and both GT_poses.txt and GT_poses_setorigin.txt. Finally, I tested the distance between GT_poses.txt and GT_poses_setorigin.txt, and it was also large. I cannot comprehend this phenomenon because performing normalization essentially involves a rigid body transformation, which should be solvable by Umeyama's method... (with scale correction). This issue did not occur when I used evo's Umeyama's method... (with scale correction) previously. This occurrence leaves me puzzled. At least the translation distance between GT_poses.txt and GT_poses_setorigin.txt should be close to 0, right? My commands and outputs are as follows:

Command:

1.

 evo_ape tum  tum_GT_poses.txt  tum_estimated_poses.txt  --plot --plot_mode xyz -v -a -s 
  evo_ape tum  tum_GT_poses.txt  tum_estimated_poses.txt   --plot --plot_mode xyz -v -a -s --pose_relation angle_deg

2.

    evo_ape tum  tum_GT_poses_setorigin.txt tum_estimated_poses.txt   --plot --plot_mode xyz -v -a -s 
  evo_ape tum  tum_GT_poses_setorigin.txt  tum_estimated_poses.txt   --plot --plot_mode xyz -v -a -s --pose_relation angle_deg

3.

 evo_ape tum  tum_GT_poses.txt tum_GT_poses_setorigin.txt  --plot --plot_mode xyz -v -a -s 
  evo_ape tum  tum_GT_poses.txt tum_GT_poses_setorigin.txt  --plot --plot_mode xyz -v -a -s --pose_relation angle_deg

**Console output:**

1.
Aligning using Umeyama's method... (with scale correction)
Rotation of alignment:
[[-0.99613627 -0.00945601  0.08731041]
 [-0.0801918   0.50324266 -0.86041624]
 [-0.03580222 -0.8640934  -0.50205656]]
Translation of alignment:
[0.20320559 1.34132595 1.15329563]
Scale correction: 1.3052250785832933
--------------------------------------------------------------------------------
Compared 30 absolute pose pairs.
Calculating APE for translation part pose relation...
--------------------------------------------------------------------------------
APE w.r.t. translation part (m)
(with Sim(3) Umeyama alignment)

       max      0.000817
      mean      0.000501
    median      0.000481
       min      0.000170
      rmse      0.000528
       sse      0.000008
       std      0.000167

..with max. time diff.: 0.01 (s) and time offset: 0.0 (s).
--------------------------------------------------------------------------------
Aligning using Umeyama's method... (with scale correction)
Rotation of alignment:
[[-0.99613627 -0.00945601  0.08731041]
 [-0.0801918   0.50324266 -0.86041624]
 [-0.03580222 -0.8640934  -0.50205656]]
Translation of alignment:
[0.20320559 1.34132595 1.15329563]
Scale correction: 1.3052250785832933
--------------------------------------------------------------------------------
Compared 30 absolute pose pairs.
Calculating APE for rotation angle in degrees pose relation...
--------------------------------------------------------------------------------
APE w.r.t. rotation angle in degrees (deg)
(with Sim(3) Umeyama alignment)

       max      0.860715
      mean      0.841904
    median      0.840947
       min      0.829709
      rmse      0.841939
       sse      21.265860
       std      0.007691

2.
ligning using Umeyama's method... (with scale correction)
Rotation of alignment:
[[-0.85042286 -0.33212524 -0.40801198]
 [-0.05391494  0.82647652 -0.56038357]
 [ 0.52332985 -0.45456506 -0.72076104]]
Translation of alignment:
[-0.01215338 -0.00098615 -0.0073351 ]
Scale correction: 3.1945023047642778
--------------------------------------------------------------------------------
Compared 30 absolute pose pairs.
Calculating APE for translation part pose relation...
--------------------------------------------------------------------------------
APE w.r.t. translation part (m)
(with Sim(3) Umeyama alignment)

       max      0.018655
      mean      0.011054
    median      0.010915
       min      0.002796
      rmse      0.011844
       sse      0.004208
       std      0.004251

Rotation of alignment:
[[-0.85042286 -0.33212524 -0.40801198]
 [-0.05391494  0.82647652 -0.56038357]
 [ 0.52332985 -0.45456506 -0.72076104]]
Translation of alignment:
[-0.01215338 -0.00098615 -0.0073351 ]
Scale correction: 3.1945023047642778
--------------------------------------------------------------------------------
Compared 30 absolute pose pairs.
Calculating APE for rotation angle in degrees pose relation...
--------------------------------------------------------------------------------
APE w.r.t. rotation angle in degrees (deg)
(with Sim(3) Umeyama alignment)

       max      150.794849
      mean      150.601538
    median      150.612843
       min      150.277719
      rmse      150.601594
       sse      680425.204015
       std      0.130155

3.

Aligning using Umeyama's method... (with scale correction)
Rotation of alignment:
[[ 0.81213839 -0.00768639 -0.58341422]
 [ 0.25132922  0.90700013  0.3379118 ]
 [ 0.52655945 -0.42106019  0.73854144]]
Translation of alignment:
[0.20701833 1.34556363 1.15596818]
Scale correction: 0.28950302320866733
--------------------------------------------------------------------------------
Compared 30 absolute pose pairs.
Calculating APE for translation part pose relation...
--------------------------------------------------------------------------------
APE w.r.t. translation part (m)
(with Sim(3) Umeyama alignment)

       max      0.006383
      mean      0.003615
    median      0.003942
       min      0.000600
      rmse      0.003940
       sse      0.000466
       std      0.001568

[[ 0.81213839 -0.00768639 -0.58341422]
 [ 0.25132922  0.90700013  0.3379118 ]
 [ 0.52655945 -0.42106019  0.73854144]]
Translation of alignment:
[0.20701833 1.34556363 1.15596818]
Scale correction: 0.28950302320866733
--------------------------------------------------------------------------------
Compared 30 absolute pose pairs.
Calculating APE for rotation angle in degrees pose relation...
--------------------------------------------------------------------------------
APE w.r.t. rotation angle in degrees (deg)
(with Sim(3) Umeyama alignment)

       max      150.031332
      mean      149.848808
    median      149.857092
       min      149.524697
      rmse      149.848865
       sse      673640.470997
       std      0.130445

Additional files: Please attach all the files needed to reproduce the error.

Please give also the following information:

Ubuntu 20.04 evo version v1.16.0 evo pkg --pyversion 3.7.11

Download lineks of input files I used above tum_GT_poses.txt tum_estimated_poses.txt tum_GT_poses_setorigin.txt

MichaelGrupp commented 5 months ago

I normalized the first frame of GT_poses.txt. To be specific, I extracted the poses of the first frame from GT_poses.txt, and then for each frame, I multiplied its poses with those of the first frame to obtain GT_poses_setorigin.txt.

If I understand correctly, your goal was to let GT_poses start at the origin. This would mean that the trajectory was transformed rigidly to the origin, without changing its shape.

But when I look at GT_poses vs GT_poses_setorigin, they have completely different shapes. This can be seen for example by plotting, comparing their path length (8cm vs 24cm), or by calculating their difference with a metric (as you also noticed, this should be zero but it's not).

So I suspect that your transformation step introduced this error and messed up your "setorigin" groundtruth trajectory. This can happen e.g. if you're using pose matrices and do the matrix multiplication in the wrong order (P*T vs T*P), but here you need to check the math of your code yourself.

the distances between the estimated poses and the ground truth poses, before and after rigid body transformation, are different

Not sure what the issue is here, this is the expected behavior. The alignment minimizes the distance. So unless you have already optimally aligned data, the distance will change compared to the unaligned case.

Running a metric to compare estimated_poses vs GT_poses gives a normal result (like "1.)" in your examples). They are similar and the alignment looks fine.

GT_poses_setorigin doesn't work well, but that's rather because of the general issues of that trajectory mentioned before, not because of alignment or the metric.

TwiceMao commented 5 months ago

@MichaelGrupp Thanks!Yes, consistent with what you mentioned. I made a mistake in obtaining GT_poses_setorigin. Specifically, I should perform the matrix multiplication P*T, but I mistakenly did T*P, where T represents GT_poses, and P represents a rigid transformation. After executing the correct matrix multiplication, the performance of GT_poses and GT_poses_setorigin became reasonable. I also have follow-up questions: (1) Do you think it's reasonable to calculate the ATE for the rotational part of camera poses? (2) Do you think it's reasonable to align camera poses using Umeyama's method first and then calculate the ATE for the rotated part of the aligned camera poses? (3) Is it only reasonable to calculate RPE for the rotational part of camera poses? (4) How is your --align_origin implemented? (5) For GT_posesandestimated_poses, their first camera poses are different, so to compute their distance, I first transform the camera pose of the first frame in GT_posesinto the identity matrix through a rigid transformation applied to all camera poses inGT_poses. A similar operation is performed on estimated_poses to ensure that its first frame is also the identity matrix. Then, I calculate the ATE for both the translation and rotation parts of GT_poses and estimated_poses after these operations. Do you think this approach is reasonable? (6) Is there any other alignment method that can align the rotational part of camera poses?

The reason for my three doubts is that in your EVO's-a -asoptions, Umeyama's method is adopted. However, Umeyama's method takes only the translational part of camera poses as input, not the rotational part. Nevertheless, its output is a similarity transformation. Moreover, this similarity transformation needs to be applied to each camera pose, causing a change in the rotation of each camera pose.

And, I have an example: there are two poses in tum format: tum_1.txt, tum_2.txt. after aligning two sets of poses using Umeyama's method, the value of the ATE for the rotation part actually increased. (1) Before alignment, the distance for the rotation part was: evo_ape tum tum_1.txt tum_2.txt -v --pose_relation angle_deg

--------------------------------------------------------------------------------
Compared 30 absolute pose pairs.
Calculating APE for rotation angle in degrees pose relation...
--------------------------------------------------------------------------------
APE w.r.t. rotation angle in degrees (deg)
(not aligned)

       max      0.754146
      mean      0.664145
    median      0.669600
       min      0.601968
      rmse      0.665402
       sse      13.282815
       std      0.040897

(2) After alignment, the distance for the rotation part became:

evo_ape tum tum_1.txt tum_2.txt -v -as --pose_relation angle_deg

..with max. time diff.: 0.01 (s) and time offset: 0.0 (s).
--------------------------------------------------------------------------------
Aligning using Umeyama's method...
Rotation of alignment:
[[ 0.51250773  0.66465388  0.54366446]
 [-0.48380259  0.74658902 -0.45666168]
 [-0.70941588 -0.02898363  0.70419391]]
Translation of alignment:
[-0.00581512  0.00968778 -0.01290893]
Scale correction: 1.0
--------------------------------------------------------------------------------
Compared 30 absolute pose pairs.
Calculating APE for rotation angle in degrees pose relation...
--------------------------------------------------------------------------------
APE w.r.t. rotation angle in degrees (deg)
(with SE(3) Umeyama alignment)

       max      61.279638
      mean      61.129781
    median      61.180745
       min      60.900483
      rmse      61.129905
       sse      112105.959311
       std      0.123085
TwiceMao commented 5 months ago

@MichaelGrupp hello,could you help me with the last questions? Big thanks~~~~