Have you ranked in KITTI Dataset?

[vkee]

Building on that comment, I noticed in the paper that it is mentioned how far the vehicle needs to travel before localization, but there are no numbers on the accuracy of the estimated trajectory. Do you have any numbers (like RMSE)? Qualitatively the results look great but I want to see quantitive results to see just how good it is.

Thanks!

[rdube]

Thanks for your interest! We currently have not tried to benchmark as our focus was primarily on developing this segment based localization technique and not on obtaining the most accurate local solution possible. It would actually be interesting to see how laser_slam can perform on this benchmark and how much can the loop closure improve the evaluated error measures. However, I am afraid that we won't have sufficient man power to work on this topic in the short or mid term.

Would you be interested to give it a try?

[vkee]

I've been looking into this but am running into a few issues. The main issue is that the ground truth files have a pose every time data is collected, whereas SegMatch only generates a pose in its trajectory every time it moves a certain threshold distance.

If the error metric was simply comparing poses individually, it would not be too much work to work around this. However, it is based on path lengths as written in the KITTI paper (Are we ready for autonomous driving? the KITTI vision benchmark suite)

I'm not quite sure how to do this since the frames at the segments may not have poses generated by SegMatch SLAM and if I grab the next pose (or previous), it will be some distance away (within the threshold distance) and possibly increase the error quite a bit.

@rdube , do you have any ideas to work around this issue?

[vkee]

Resolved the issues but am getting pretty poor quantitative performance (qualitatively, the performance looks okay) on par with some of the bottom performing methods on the KITTI Odometry benchmark (http://www.cvlibs.net/datasets/kitti/eval_odometry.php) so far. Here are the plots from running on sequence 6 in the dataset.

Planning on playing with adding in odometry measurements and playing with parameters. @rdube any thoughts?

[rdube]

@vkee very nice to see that you are working on this! For this experiment are you using the LiDAR data only? Can you share the scripts that you used for this experiment? Calibrating the noise models should definitely be considered. You could also have a look in the input filters and ICP parameters. https://github.com/ethz-asl/segmatch/blob/master/laser_mapper/launch/kitti/input_filters_outdoor.yaml

Also how do the metrics turn out when you disable the loop closures? It could be that adding the loop closures actually make these metrics look worst if the noise models are not tuned.

For these plots did you solve the issue you mentioned in your previous post? Translation errors of 50%+ seem a bit excessive!

Concerning "SegMatch only generates a pose in its trajectory every time it moves a certain threshold distance", you can disable this by setting the following value to zero: https://github.com/ethz-asl/segmatch/blob/master/laser_mapper/launch/kitti/kitti_loop_closure.yaml#L12

As you mentioned, you should interpolate between poses if the ground truth measurements lie between two nodes in the pose graph.

For your information we are working on an updated version of SegMatch which yields better place recognition performances. The LiDAR odometry part did not change much though as getting this local accuracy was not the main focus of this work. To be clear, what you are evaluating here is the performance of laser_slam and not of SegMatch, the latter being a place recognition algorithm and not a SLAM algorithm

Still I am happy to see some quantitative results there and am sure that it is possible to parametrize it better so that it competes with other methods (some of which probably also have the same ICP - pose-graph backend).

I hope that helps!

[vkee]

@rdube, yes I am currently using only the LiDAR data. I am pretty much using the latest code of SegMatch and laser_slam with minor changes just for logging (here are my forks - https://github.com/vkee/laser_slam, https://github.com/vkee/segmatch) with the "out of the box" parameters from kitti_loop_closure.launch. Here is my repo with my modified evaluation code from the KITTI Odometry Benchmark (https://github.com/vkee/kitti_eval). Calibrating the noise models sounds good - do you have any recommendation on where to start and values to try?

I will try that and let you know - although the loop closures help greatly as the drift quickly compounds (see this screencapture on the evaluated dataset - https://www.dropbox.com/s/wzsq3yyy9wbp2h9/2011_09_30_drive_0020_icp_only.ogv?dl=0).

Yes, and yes they do - I was very surprised. Here is a bit of what I did for evaluation. Each time a laser scan comes in, I store whether a pose is stored (if the sensor has moved enough and therefore the scan is processed) and its corresponding index in the output trajectory. Once laser_slam is done processing a dataset, I store the optimized trajectory as well as the data structure denoting whether a pose was stored when each laser scan was received as well as the index to the pose in the output trajectory if it was stored. KITTI has the ground truth pose at each laser scan so in the evaluation code, I use the data structure to only evaluate the poses that were stored. I think this solves the issue I mentioned previously.

Okay I will try that.

I don't think this is necessary due to my approach to evaluating as stated above.

You are correct, but since SegMatch is used and greatly improves the performance, I am somewhat evaluating SegMatch too I think?

I think so too.

This is very helpful!

[dotPiano]

Hi @vkee! From your video it looks like many possible loop closures are not detected. A quick hint: check what value you are using for the parameter min_time_between_segment_for_matches_s (https://github.com/ethz-asl/segmatch/blob/master/laser_mapper/launch/kitti/kitti_loop_closure.yaml#L61).

That threshold is used to forbid matches between the source map and segments created in the last min_time_between_segment_for_matches_s seconds. It is necessary to avoid loop closures with segments that are still in the source map and are thus incomplete.

I think in your case you could safely go down to 20s or even less, it should increase the number of loop closures and the quality of the target map!

[vkee]

@dotPiano I tried that and while it seemed to drift less during run time (due to more loop closures earlier on), the resulting trajectory and map is about the same. I think I will try incorporating odometry information.

[vkee]

@rdube the parameters definitely need to get tuned - I ran with loop closures and ICP turned off (only odom) and I got the best performance so far (still RMSE of 5m at times though). Do you have any recommendations on where to start with tuning parameters?

Thanks!

[rdube]

@vkee note that if you are using the bag and launch files that we provided, the "odometry" information actually comes from the GPS (integrated with IMU I believe). As far as I understand from the KITTI VO benchmarking dataset, there is no odometry information provided.

Therefore an error of 5m is surprising given that this is using the GPS data. I would try to investigate this before going further with tuning parameters. Note that when using GPS data only, laser_slam will not have to perform ICP and will process incoming scans at a much higher frequency. That could hint towards the differences in performance metrics.

As for calibrating the noise models you could have a look at the flowing paper to get some insight about evaluating the covariance of ICP: http://www.mva-org.jp/Proceedings/2015USB/papers/14-27.pdf

[vkee]

@rdube I think you are right. What tool did you use to generate the bag? https://github.com/ethz-asl/kitti_to_rosbag?

I was also very surprised given the resulting generated trajectory and map. I looked into it more and found some issues with the rosbag tool I was using (https://github.com/ethz-asl/kitti_to_rosbag), I filed an issue here which discusses the tf issue (https://github.com/ethz-asl/kitti_to_rosbag/issues/5).

Thanks for the paper reference, I will definitely look into that once I resolve this issue!

[rdube]

@vkee yes that is the correct tool we used.

[vkee]

@rdube you may want to regenerate and upload the bag files you provide with this tool as the tf issue was resolved. Using just GPS data now (with the fixed bags), I am getting these much more reasonable results.

However, I am still a bit confused because I got about the same results (a few less peaks) when recording just the tf poses (pure odometry trajectory - no SegMatch SLAM or ICP involved). I would expect that to be near 0 RMSE because the ground truth is supposedly from the same GPS/IMU data.

[rdube]

@vkee thanks for pointing this out! As discussed there might be a difference between the data that the KITTI benchmark uses and the provided GPS + IMU data. For this issue the people responsible of the KITTI benchmark should be of better support. Is there something more to address here or can we close this issue?

[vkee]

I think this issue can be closed.

For those who may be interested, I think the GPS + IMU data used to generate the tfs with the kitti to rosbag tools (https://github.com/ethz-asl/kitti_to_rosbag or https://github.com/tomas789/kitti2bag) can be assumed to be the ground truth.