Bug in Colab demo with our own dataset

[lck666666]

I want to construct my own sequnce data captured by a camera. I use the script in Colab/demo.ipynb

We get the following warning when the code runs here.

model = reconstruction.main(sfm_dir, images, sfm_pairs, features, matches, image_list=references)
fig = viz_3d.init_figure()
viz_3d.plot_reconstruction(fig, model, color='rgba(255,0,0,0.5)', name="mapping")
fig.show()

In fact, when we run the dataset of 10 images in demo, the result is good. When we run our own dataset, if the number of images in mapping is small, such as 10 or 20, there is no problem. However, once the number of images increases, such as 50 and 60, the following warning will occur.

[2022/01/10 08:36:03 hloc INFO] Performing geometric verification of the matches...
[2022/01/10 08:36:52 hloc INFO] Running 3D reconstruction...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0110 08:36:57.561012  2854 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: Eigen failure. Unable to perform dense Cholesky factorization.
W0110 08:36:57.568457  2854 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: Eigen failure. Unable to perform dense Cholesky factorization.
W0110 08:36:57.569886  2854 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: Eigen failure. Unable to perform dense Cholesky factorization.
W0110 08:36:58.014516  2854 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: Eigen failure. Unable to perform dense Cholesky factorization.
W0110 08:36:58.022526  2854 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: Eigen failure. Unable to perform dense Cholesky factorization.
W0110 08:37:05.537235  2854 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: Eigen failure. Unable to perform dense Cholesky factorization.
[2022/01/10 08:37:34 hloc INFO] Reconstructed 1 model(s).
[2022/01/10 08:37:34 hloc INFO] Largest model is #0 with 62 images.
[2022/01/10 08:37:34 hloc INFO] Reconstruction statistics:
Reconstruction:
    num_reg_images = 62
    num_cameras = 62
    num_points3D = 5456
    num_observations = 26143
    mean_track_length = 4.79161
    mean_observations_per_image = 421.661
    mean_reprojection_error = 1.0953
    num_input_images = 64

We suspect that this warning is caused by the density of some of the image feature points, but our data set seems to be small and the images are not unusual.

But this still ends up with the corresponding bin file under sfm_dir. As shown in the result, the num_input_images is 64, the num_reg_images is 62. This difference will cause an error in the following code:

import pycolmap
from hloc.localize_sfm import QueryLocalizer, pose_from_cluster

camera = pycolmap.infer_camera_from_image(images / query)
ref_ids = [model.find_image_with_name(r).image_id for r in references]
conf = {
    'estimation': {'ransac': {'max_error': 12}},
    'refinement': {'refine_focal_length': True, 'refine_extra_params': True},
}
localizer = QueryLocalizer(model, conf)
ret, log = pose_from_cluster(localizer, query, camera, ref_ids, features, matches)

print(f'found {ret["num_inliers"]}/{len(ret["inliers"])} inlier correspondences.')
visualization.visualize_loc_from_log(images, query, log, model)

Because demo.ipynb defined reference as all mapping images in the beginning of the code as references = [str(p.relative_to(images)) for p in (images / 'mapping/').iterdir()], the reconstruction will not necessarily use all references images. The demo script has no errors because num_input_images = num_reg_images = 10. So, we think the references variable should be reassigned after getting the sfm model. Otherwise if num_input_images != num_reg_images, there will be an attribute error.

So we have two questions,

1. Does this warning have a big impact on reconstruction?
2. What caused the reconstruction not to use all images, is that normal?

Looking forward to your reply! Thanks sincerely.

[sarlinpe]

1. This warning is fine, it should not impact the reconstruction.

2. Nice catch. I have updated the Colab notebook to get the list of images that are successfully registered:

   references_registered = [model.images[i].name for i in model.reg_image_ids()]

   and then use this subset to restrict the matching and the localization. Does this solve your problem?

3. In general, it can indeed happen that some images are not registered into the 3D model. You can easy get their names with set(references) - set(references_registered) and inspect them.

[lck666666]

2. Nice catch. I have updated the Colab notebook to get the list of images that are successfully registered:

references_registered = [model.images[i].name for i in model.reg_image_ids()]

and then use this subset to restrict the matching and the localization. Does this solve your problem?

Yes, it solved. Thanks!