Question Regarding Loop Closure Plugin: 'eloc' function

[amitfishy]

Hello,

I've been trying to use the loop closure plugin (from src/maplab/console-plugins/loop-closure-plugin/loop-closure-plugin.cc) - particularly the evaluate_localization (eloc) command.

Following the cla data (floor f-j), I built a map following the guides mentioned and the calibration file from the data. What I would like to do is take data from the localization data (as query images frame by frame) and find out where they belong in the created map (I would like to get the frame data from the rosbag for the matched frame). I understand the matching is done in a 2D-3D way as mentioned in the paper, but is there a way to pin-point the data from query image to rosbag files (used for creating the map)?

Here are the steps I follow:

1. Build maps and optimize as mentioned in tutorials for floor f,g,h,j.
2. Use ROVIOLI in VIO mode using the mentioned tutorial.
3. I load the joint map of floors f,g,h,j in the maplab console.
4. I load the cla_floor_f_v2_for_localization map just created in point 2.
5. I run a join_all_maps --target_map_key cla_merged and then sbk (side question, how do you select a non-default sbk?)
6. Then finally I run eloc

I0628 15:15:08.115725 16864 maplab-console.cc:100] RVIZ visualization initialized!
maplab <>:/$ load --map_folder '/data/amit_data/maplab/floor_data/maps_created/cla_full'
Progress: 0:100.0% [#############################################################################]  
--->Complete!
Progress: 0:100.0%  1:100.0%  2:100.0%  3:100.0%  4:100.0%  5:100.0%                                
--->Complete!
I0628 15:15:14.764834 16864 vi-map-serialization.cc:590] Loaded VIMap from "/data/amit_data/maplab/floor_data/maps_created/cla_full".

maplab <cla_full>:/$ load --map_folder '/data/amit_data/maplab/floor_data/maps_created/cla_test'
Progress: 0:100.0% [#############################################################################]  
--->Complete!
Progress: 0:100.0%  1:100.0%  2:100.0%  3:100.0%  4:100.0%  5:100.0%                                
--->Complete!
I0628 15:15:19.764120 16864 vi-map-serialization.cc:590] Loaded VIMap from "/data/amit_data/maplab/floor_data/maps_created/cla_test".

maplab <cla_test>:/$ join_all_maps --target_map_key cla_merged
I0628 15:15:36.314869 16864 vi-map-basic-plugin.cc:576] Merging "cla_full" into "cla_test".
I0628 15:15:36.314921 16864 vi-map-basic-plugin.cc:591] Merged 2 maps together into the target map "cla_merged".

maplab <cla_merged>:/$ ms
\Mission statistics: 

Mission 0:                  2c951998ffffffff0f00000000000000    Viwls
    NCamera Sensor:         412eab8e4058621f7036b5e765dfe813
    IMU Sensor:             c63aecb41bfdfd6b7e1fac37c7cbe7bf
    Vertices:               1849                     
    Landmarks:              47554                    
    Landmarks by first observer backlink:                            
    Camera 0:               47554 (g:10727 b:36827 u:0)
    Observations:           469147                   
    Num edges by type:                               
    IMU:                    1848                     
    Wheel-odometry:         0                        
    Loop-closure:           0                        
    Distance travelled [m]: 204.786328               
    Start to end time:      2018_06_28_15_15_39 to 2018_06_28_15_15_39
    T_G_M                   unknown                  

Mission 1:                  5d086175000000000f00000000000000    Viwls
    NCamera Sensor:         412eab8e4058621f7036b5e765dfe813
    IMU Sensor:             c63aecb41bfdfd6b7e1fac37c7cbe7bf
    Vertices:               398                      
    Landmarks:              36960                    
    Landmarks by first observer backlink:                            
    Camera 0:               36960 (g:13820 b:23140 u:0)
    Observations:           91164                    
    Num edges by type:                               
    IMU:                    397                      
    Wheel-odometry:         0                        
    Loop-closure:           0                        
    Distance travelled [m]: 247.172438               
    Start to end time:      2018_06_28_15_15_39 to 2018_06_28_15_15_39
    T_G_M                   unknown                  

Mission 2:                  e6ce16bfffffffff0f00000000000000    Viwls
    NCamera Sensor:         412eab8e4058621f7036b5e765dfe813
    IMU Sensor:             c63aecb41bfdfd6b7e1fac37c7cbe7bf
    Vertices:               446                      
    Landmarks:              41372                    
    Landmarks by first observer backlink:                            
    Camera 0:               41372 (g:15263 b:26109 u:0)
    Observations:           103138                   
    Num edges by type:                               
    IMU:                    445                      
    Wheel-odometry:         0                        
    Loop-closure:           0                        
    Distance travelled [m]: 267.430769               
    Start to end time:      2018_06_28_15_15_39 to 2018_06_28_15_15_39
    T_G_M                   unknown                  

Mission 3:                  34844133000000000f00000000000000    Viwls
    NCamera Sensor:         412eab8e4058621f7036b5e765dfe813
    IMU Sensor:             c63aecb41bfdfd6b7e1fac37c7cbe7bf
    Vertices:               500                      
    Landmarks:              50947                    
    Landmarks by first observer backlink:                            
    Camera 0:               50947 (g:19660 b:31287 u:0)
    Observations:           122353                   
    Num edges by type:                               
    IMU:                    499                      
    Wheel-odometry:         0                        
    Loop-closure:           0                        
    Distance travelled [m]: 298.227086               
    Start to end time:      2018_06_28_15_15_39 to 2018_06_28_15_15_39
    T_G_M                   unknown                  

Mission 4:                  65dcd74f000000000f00000000000000    Viwls
    NCamera Sensor:         412eab8e4058621f7036b5e765dfe813
    IMU Sensor:             c63aecb41bfdfd6b7e1fac37c7cbe7bf
    Vertices:               342                      
    Landmarks:              27055                    
    Landmarks by first observer backlink:                            
    Camera 0:               27055 (g:9967 b:17088 u:0)
    Observations:           71316                    
    Num edges by type:                               
    IMU:                    341                      
    Wheel-odometry:         0                        
    Loop-closure:           0                        
    Distance travelled [m]: 221.613279               
    Start to end time:      2018_06_28_15_15_39 to 2018_06_28_15_15_39
    T_G_M                   unknown                  

Accumulated statistics over all missions:
    Number of missions:     5                        
    Vertices:               3535                     
    Landmarks:              203888 (g:69437 b:134451 u:0)
    Observations:           857118                   
    Distance travelled [m]: 1239.229901              
    Duration [s]:           1108.550000              

maplab <cla_merged>:/$ sbk
W0628 15:15:47.808974 16864 vi-map.cc:174] Mission ID not set or invalid, taking 1st mission ID: e6ce16bfffffffff0f00000000000000

maplab <cla_merged>:/$ eloc
W0628 15:16:10.593041 16864 vi-map.cc:174] Mission ID not set or invalid, taking 1st mission ID: e6ce16bfffffffff0f00000000000000
I0628 15:16:10.626930 16864 vi-localization-evaluator.cc:52] Will query against 162516 landmarks.
I0628 15:16:12.351209 16864 vi-localization-evaluator.cc:58] Evaluating the localizations.
W0628 15:16:12.355253 16916 localization-evaluator.cc:60]   Ransac failed:0 matches, 0 inliers.
W0628 15:16:12.355607 16916 localization-evaluator.cc:68]   Couldn't localize af89..0000
W0628 15:16:12.356052 16917 localization-evaluator.cc:60]   Ransac failed:0 matches, 0 inliers.
.
.
.
W0628 15:16:12.994571 16914 localization-evaluator.cc:60]   Ransac failed:14 matches, 0 inliers.
W0628 15:16:12.994603 16914 localization-evaluator.cc:68]   Couldn't localize 1732..0000
W0628 15:16:12.999505 16914 localization-evaluator.cc:68]   Couldn't localize 1ebd..0000
W0628 15:16:13.004051 16914 localization-evaluator.cc:68]   Couldn't localize e053..0000
W0628 15:16:13.004686 16914 localization-evaluator.cc:60]   Ransac failed:0 matches, 0 inliers.
W0628 15:16:13.004706 16914 localization-evaluator.cc:68]   Couldn't localize a74c..0000
I0628 15:16:13.004915 16864 vi-localization-evaluator.cc:62] Recall: 0.0403587

So I'm unable to figure out how to set the query and the database properly wrt these missions. Also I know that e6ce16bfffffffff0f00000000000000 refers to one of the floor runs here and the localization run is supposed to be 2c951998ffffffff0f00000000000000 in which case I get a Recall of 0 at the end (I was able to set it as sbk by changing the order of the operations).

Thank you for your time.

[amitfishy]

So,

I was able to get a seemingly proper result by doing the following:

load --map_folder '/data/amit_data/maplab/floor_data/maps_created/cla_test'

load --map_folder '/data/amit_data/maplab/floor_data/maps_created/cla_full'

join_all_maps --target_map_key cla_merged

align_for_eloc       //This takes 2c951998ffffffff0f00000000000000 as default because we loaded it first

eloc       //This takes 2c951998ffffffff0f00000000000000 as default because we loaded it first

Output:

I0628 15:39:44.669945 17208 vi-localization-evaluator.cc:62] Recall: 0.24662

So from what I understand this recall is the % of frames that localized correctly between the query mission (the mission that gets set by default on running align_for_eloc and eloc) and the map that was created. Could someone please confirm this?

Please also have a look at my question regarding extracting further information from the bagfile frame identified by the query image.

Thanks

[dymczykm]

Hi @amitfishy,

I'll try to answer your questions one by one:

• The only way to associate the keyframes back with the rosbag is to use the timestamp. You can find it in the VisualNFrame that is stored (as a pointer actually) inside the Vertex.
• sbk can be used with a -map_mission= flag, e.g. sbk --map_mission=7d8bb3e8145b3c150d00000000000000.
• Same for eloc, you can select the query mission using the same -map_mission flag (see the code here).
• Yes exactly, this is indeed recall as you describe. It is expressed as a percentage of frames that got correctly localized. The distance threshold can be varied using --benchmark_position_error_threshold, by default it is 0.05m.

Hope this helps.

[amitfishy]

Thank you for your response! Code reference was nice to have as well.

If you have the time could you please answer these as well?

1. Regarding the align_for_eloc or any other align functions (aam) which align a mission with the base frame, Do these use loop closures to put the different mission frames in the same co-ord frame? My confusion comes from the separate use of lc which I understand from here, just merges the loop closures that were found during the alignment phase.
2. I am actually trying to benchmark the performance of the loop closure. Something of the sort of AUC metric. But it seems we don't really have a GT reference for our localization run over the built map, so we can only benchmark maplab loop closures at the 100% precision assumption (What I mean to say is, The recall is the algorithm performance based on assuming all loop closures are correct - 100% precision - What is the ground truth to validate correct loop closures?). Do we not need a GT because all detected loop closures are guaranteed to be correct?

[dymczykm]

@amitfishy

1. Yes, align_for_eloc uses the loop closure engine, but only temporarily merges matching landmarks and then reverts the changes. We wanted to revert them to make sure eloc works in a fair way, e.g. no landmarks are merged before the evaluation.
2. This is a tricky question. The loop closures are not guaranteed to be correct, but with the default settings of PnP RANSAC pose estimation (15+ inliers, 2-3px reprojection error) the chance of getting a false positive is rather small. In some of our papers, we have assumed that the trajectories after align_for_eloc are well aligned and you can do the following:
   • get an optimized (using align_for_eloc) pose of the keyframe expressed in the common frame for both missions,
   • compare it to the pose estimated using the loop closure in the common frame with the other mission. Our experience shows that these should generally match well. You can check how eloc does it (it actually checks the pose error).

[amitfishy]

Thanks for the information, it helps a lot.