search

naver / kapture-localization

Provide mapping and localization pipelines based on kapture format
BSD 3-Clause "New" or "Revised" License
268 stars 42 forks source link

Some images are not localized using localSfM #31

Closed JSP-ywu closed 2 years ago

JSP-ywu commented 2 years ago

Hi! I tried to make prediction with kapture_pipeline_image_retrieval_benchmark.py. I gave 1200 test images to predict but got only 940 images (All images are from my custom dataset). All features are from R2D2 and AP-GeM. Here is the failure log.

INFO    ::root: mapping and localization for 20220825-125110.603-11~12-Tr/stage_1/images/1170_1170_frame_000544.png
INFO    ::kapture: deleting already existing /home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/colmap.db
INFO    ::kapture: deleting already existing /home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/reconstruction
INFO    ::kapture: deleting already existing /home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/priors_for_reconstruction
INFO    ::colmap_build_map: Using precomputed keypoints and matches
INFO    ::colmap_build_map: Step 1: Export kapture format to colmap
INFO    ::colmap: registering 1 sensors (cameras) in database...
INFO    ::colmap: registering 20 images in database...
INFO    ::colmap: registering 20 keypoints in database...
INFO    ::colmap: registering 190 matches in database...
INFO    ::colmap_build_map: Step 2: Run geometric verification - skipped
INFO    ::colmap_build_map: Step 3: Exporting priors for reconstruction.
INFO    ::colmap: creating colmap cameras.txt
INFO    ::colmap: creating colmap images.txt
INFO    ::root: creating colmap points3D.txt
INFO    ::colmap_build_map: Step 4: Triangulation
INFO    ::colmap: ['colmap', 'point_triangulator', '--database_path', '/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/colmap.db', '--image_path', '/home/pjs4073/kapture_localsfm_h4tech/kapture_inputs/proxy_map_plus_query_gv/sensors/records_data', '--input_path', '/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/priors_for_reconstruction', '--output_path', '/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/reconstruction']

==============================================================================
Loading model
==============================================================================

==============================================================================
Loading database
==============================================================================

Loading cameras... 1 in 0.000s
Loading matches... 190 in 0.000s
Loading images... 20 in 0.003s (connected 20)
Building correspondence graph... in 0.003s (ignored 0)

Elapsed time: 0.000 [minutes]

==============================================================================
Triangulating image #1 (0)
==============================================================================

  => Image sees 0 / 1457 points
  => Triangulated 0 points

==============================================================================
Triangulating image #2 (1)
==============================================================================

  => Image sees 0 / 655 points
  => Triangulated 1 points

==============================================================================
Triangulating image #3 (2)
==============================================================================

  => Image sees 3 / 1433 points
  => Triangulated 5 points

==============================================================================
Triangulating image #4 (3)
==============================================================================

  => Image sees 2 / 844 points
  => Triangulated 9 points

==============================================================================
Triangulating image #5 (4)
==============================================================================

  => Image sees 9 / 1407 points
  => Triangulated 1 points

==============================================================================
Triangulating image #6 (5)
==============================================================================

  => Image sees 14 / 506 points
  => Triangulated 16 points

==============================================================================
Triangulating image #7 (6)
==============================================================================

  => Image sees 0 / 930 points
  => Triangulated 0 points

==============================================================================
Triangulating image #8 (7)
==============================================================================

  => Image sees 0 / 1464 points
  => Triangulated 0 points

==============================================================================
Triangulating image #9 (8)
==============================================================================

  => Image sees 20 / 1358 points
  => Triangulated 0 points

==============================================================================
Triangulating image #10 (9)
==============================================================================

  => Image sees 0 / 638 points
  => Triangulated 0 points

==============================================================================
Triangulating image #11 (10)
==============================================================================

  => Image sees 0 / 636 points
  => Triangulated 0 points

==============================================================================
Triangulating image #12 (11)
==============================================================================

  => Image sees 0 / 1488 points
  => Triangulated 0 points

==============================================================================
Triangulating image #13 (12)
==============================================================================

  => Image sees 6 / 648 points
  => Triangulated 24 points

==============================================================================
Triangulating image #14 (13)
==============================================================================

  => Image sees 2 / 461 points
  => Triangulated 5 points

==============================================================================
Triangulating image #15 (14)
==============================================================================

  => Image sees 9 / 688 points
  => Triangulated 0 points

==============================================================================
Triangulating image #16 (15)
==============================================================================

  => Image sees 0 / 961 points
  => Triangulated 0 points

==============================================================================
Triangulating image #17 (16)
==============================================================================

  => Image sees 0 / 557 points
  => Triangulated 0 points

==============================================================================
Triangulating image #18 (17)
==============================================================================

  => Image sees 12 / 970 points
  => Triangulated 0 points

==============================================================================
Triangulating image #19 (18)
==============================================================================

  => Image sees 44 / 500 points
  => Triangulated 0 points

==============================================================================
Triangulating image #20 (19)
==============================================================================

  => Image sees 0 / 1467 points
  => Triangulated 0 points

==============================================================================
Retriangulation
==============================================================================

  => Merged observations: 0
  => Completed observations: 0

==============================================================================
Bundle adjustment
==============================================================================

iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time
   0  3.040423e+05    0.00e+00    2.77e+02   0.00e+00   0.00e+00  1.00e+04        0    1.61e-02    1.64e-02
   1  2.355396e+05    6.85e+04    2.07e+01   7.38e+05   9.47e-01  3.00e+04        0    3.20e-02    4.84e-02
   2  2.343848e+05    1.15e+03    1.62e+00   4.71e+05   8.71e-01  5.08e+04        0    2.87e-04    4.89e-02
   3  2.343163e+05    6.85e+01    2.15e-01   2.76e+05   7.79e-01  6.16e+04        0    2.78e-04    4.92e-02

Bundle adjustment report
------------------------
    Residuals : 456
   Parameters : 333
   Iterations : 4
         Time : 0.0493292 [s]
 Initial cost : 25.8217 [px]
   Final cost : 22.6683 [px]
  Termination : Convergence

  => Merged observations: 0
  => Completed observations: 0
  => Filtered observations: 215
  => Changed observations: 0.942982

==============================================================================
Bundle adjustment
==============================================================================

iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time
   0  3.524065e+01    0.00e+00    3.16e-06   0.00e+00   0.00e+00  1.00e+04        0    2.86e-05    1.77e-04

Bundle adjustment report
------------------------
    Residuals : 26
   Parameters : 18
   Iterations : 1
         Time : 0.000349139 [s]
 Initial cost : 1.16422 [px]
   Final cost : 1.16422 [px]
  Termination : Convergence

  => Merged observations: 0
  => Completed observations: 0
  => Filtered observations: 0
  => Changed observations: 0.000000

==============================================================================
Extracting colors
==============================================================================

WARNING ::colmap_localize: Input data contains trajectories: they will be ignored
INFO    ::kapture: deleting already existing /home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/colmap.db
INFO    ::kapture: deleting already existing /home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/reconstruction
20it [00:00, 355449.49it/s]
INFO    ::colmap_localize: Step 1: Add precomputed keypoints and matches to colmap db
INFO    ::colmap_localize: Step 2: Run geometric verification - skipped
INFO    ::colmap_localize: Step 3: Run image_registrator
INFO    ::colmap: ['colmap', 'image_registrator', '--database_path', '/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/colmap.db', '--input_path', '/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/reconstruction', '--output_path', '/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/reconstruction', '--Mapper.ba_refine_focal_length', '0', '--Mapper.ba_refine_principal_point', '0', '--Mapper.ba_refine_extra_params', '0', '--Mapper.min_num_matches', '4', '--Mapper.init_min_num_inliers', '4', '--Mapper.abs_pose_min_num_inliers', '4', '--Mapper.abs_pose_min_inlier_ratio', '0.05', '--Mapper.ba_local_max_num_iterations', '50', '--Mapper.abs_pose_max_error', '20', '--Mapper.filter_max_reproj_error', '12']

==============================================================================
Loading database
==============================================================================

Loading cameras... 2 in 0.000s
Loading matches... 210 in 0.001s
Loading images... 21 in 0.003s (connected 21)
Building correspondence graph... in 0.004s (ignored 0)

Elapsed time: 0.000 [minutes]

==============================================================================
Registering image #21 (21)
==============================================================================

  => Image sees 3 / 1303 points
INFO    ::colmap_localize: Step 4: Export reconstruction results to txt
INFO    ::colmap: ['colmap', 'model_converter', '--input_path', '/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/reconstruction', '--output_path', '/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/reconstruction', '--output_type', 'TXT']
DEBUG   ::colmap: importing from database "/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/colmap.db"
DEBUG   ::colmap: loading colmap database /home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/colmap.db
DEBUG   ::colmap: parsing cameras in database.
INFO    ::colmap: parsing cameras  ...
DEBUG   ::colmap: parsing images and trajectories in database.
INFO    ::root: parsing images ...
21it [00:00, 11695.71it/s]
DEBUG   ::colmap: importing from reconstruction "/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/reconstruction"
DEBUG   ::colmap: loading colmap reconstruction from:
        "/home/pjs4073/kapture_localsfm_h4tech/local_sfm/colmap/registered/reconstruction"
DEBUG   ::colmap: loading colmap reconstruction skipping Observations, Points3d, Keypoints
DEBUG   ::root: parsing cameras from:
        "cameras.txt"
DEBUG   ::root: loading images from:
        "images.txt"
INFO    ::root: 20220825-125110.603-11~12-Tr/stage_1/images/1170_1170_frame_000544.png was not localized
DEBUG   ::kapture: saving sensors ...
DEBUG   ::kapture: saving trajectories ...
DEBUG   ::kapture: wrote          938 <class 'kapture.core.Trajectories.Trajectories'> in 0.015 seconds
DEBUG   ::kapture: saving records_camera ...
DEBUG   ::kapture: wrote        1 065 <class 'kapture.core.Records.RecordsCamera'> in 0.002 seconds
DEBUG   ::kapture: saving keypoints : r2d2_WASF_N8_big ...
DEBUG   ::kapture: saving descriptors : r2d2_WASF_N8_big ...
DEBUG   ::kapture: saving global_features : Resnet101-AP-GeM-LM18 ...
INFO    ::kapture: Saved in 0.018 seconds in "/home/pjs4073/kapture_localsfm_h4tech/local_sfm/localized"
mhumenbe commented 2 years ago

It seems like the resulting local map only consists of a few 3D points that are not enough for registration. Have you tried to increase the number of retrieved images?

JSP-ywu commented 2 years ago

I set the top-k parameter to 20. The parameter is recommended one. So I think if this problem is caused bythe lack of 3D points, should I increase the number of keypoints? Because the images in my dataset have small size, 640x480, I set the number of keypoints to 3k.

mhumenbe commented 2 years ago

If you increase the number of retrieved images, you might also get more matches and, thus, more 3D points. So you could try to use, say, 50. 640x480 is quite low. But sure, try to increase the number of keypoints as well.

It also depends on the composition of your dataset. Local_sfm works well if the retrieved images show the same area from different locations and view points. However, local feature matching still has to work.

JSP-ywu commented 2 years ago

I'll try it! Thanks for your advice!