search

YibinWu / LIO-EKF

Maybe the simplest LiDAR-inertial odometry that one can have.
MIT License
151 stars 17 forks source link

Question on paper result and contribution claim #2

Closed snakehaihai closed 4 months ago

snakehaihai commented 4 months ago

Hi In your work, you indicate" In sum, we make three key claims: LIO-EKF (i) is on par with the state-of-the-art LIO systems in terms of estimation accuracy, (ii) can provide an accurate pose estimate in different environments and vehicle motion profiles, (iii) can provide pose estimates at close-to-IMU frame rate. These claims are backed up by our experimental evaluation"

And, In your work, you also indicate that you were inspired by KISS ICP. Why are you not comparing with KISS ICP or CT ICP for the NEWER COLLEGE DATASET?

Since you submitted it to ICRA 2024, at least I would expect an open-source SOTA from ICRA 2023 comparison like DLIO, which happens to be also run on the Newer Colledge dataset. Why was it not included? Why other people can run LIO SAM in the same dataset and you can not? Why other people can get 0.3-0.4 for fast lio2 and you get 5-6m? Is the measurement have some trick? like using end to end point measurement and not ATE? Did you align the sequence?

image Above is author claimed result

image Above is the DLIO claimed result image Above is the LOG-LIO Claimed result. image This is result from Fast and Robust Lidar-Inertial Odometry by Tightly-Coupled Iterated Kalman Smoother and Robocentric Voxels

If all of them consistently getting better result than you for the baseline methods, that means something is wrong with your estimation

brytsknguyen commented 4 months ago

The main contribution of the paper is an implementation that is more accurate and fast, so I'd recommend you include more state of the art LIO methods: DLIO, SLICT, Traj-LIO, CLIC in your experiments. Besides, some LO methods such as KISS-ICP, CT-ICP can also be included to reinforce the advantage of LIO over LO.

brytsknguyen commented 4 months ago

Hi,

How is the ATE calculated here? This number is too big. In SLICT and CLIC they are much smaller image

This is CLIC result image

This is the SLICT result image

DLIO has similar results with SLICT image

YibinWu commented 4 months ago

Hey, thanks for your comments.

  1. Yes. We have compared with KISS-ICP in the old version. Here is the table. We removed this because we thought it was unfair since KISS-ICP and other LO systems didn't have an IMU. In addition, the advantages of adding IMU into a LO are apparent. Therefore, we didn't report it. image

  2. For the metrics, we had the same findings. We used the metric implemented in KISS-ICP to get the numbers. We also tried evo, and the numbers were smaller for all systems. Here is an example.
    image image We didn't pay too much for selecting the metric since we thought using the same one for every system was fair. In addition, it doesn't affect our claim that it achieves accuracy similar to the others. Also, in the DLIO tables you showed, Fast-LIO2 reports ~42 ms average processing time in newerCollege dataset, but in our table, it is only ~9 ms.

  3. We failed to run LIO-SAM with the newerCollege dataset because it requires attitude information from the IMU, which is not the case in the dataset. Therefore, we didn't get the results. We have explained this in the paper. image

  4. This work wants to show that with a simple design, LIO-EKF can achieve higher efficiency while not losing much accuracy. We focused more on computation efficiency instead of challenging all the SOTA in terms of accuracy. Therefore, we selected the two most representative LIO systems. For example, in the DLIO tables you showed, it was also only compared with these two LIO systems.

Hope this helps.

YuePanEdward commented 4 months ago

The difference between the two evaluation metrics is due to the additional multiplication of the rotation error to the reference position.

As shown in Equation (23) of the paper A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry:

image

As also mentioned in this issue: https://github.com/uzh-rpg/rpg_trajectory_evaluation/issues/58

Evo and other evaluation libraries use the one without the multiplication of the rotation error while kiss-icp and lio-ekf uses the one with the multiplication of the rotation error.

snakehaihai commented 4 months ago

Thanks for the prompt response.

OK, I think I get the difference. EVO shows better results by giving additional scale adjustment whereas KISSICP does not

For speed wise, i do concur with you image

Since for EVO, it has similar result with others. then i guess that clears the cloud.

Then, I guess it did similar contribution as FLOAM. Just that some sequences are good, some sequences are bad. it is expected for most of existing LIO system

If possible, align all scales and rotations for all sequences so that your proposed system result can be directly compared with others.

One advice. Try to bolden the best result and second best in underline.