Segfault during triangulation

ThomasParistech commented 2 years ago

I'm trying to refine a Pointcloud extracted using ArCore on my phone, and I get a Segmentation fault error at the end of the triangulation part of PixSfM.

As an input, I provide:

A folder of images
A binary colmap model (cameras.bin, images.bin, points3D.bin): The pointcloud is noisy and used only for the _pairs_fromcovisibility method (In images.bin, point3D_ids make sense but I simply put dummy zeros xys for the 2d features since features should be recomputed by hloc) but the intrinsics/extrinsics estimate from arcore are quite good and should be used as priors

and I want to refine the poses and the pointcloud.

My pipeline is the following: 1) hloc.extract_features 2) hloc.pairs_from_covisibility 3) hloc.match_features 4) PixSfM.triangulation

I use the low memory configuration file, because it crashes otherwise.

Here's my code:

from pathlib import Path
from hloc import extract_features, match_features, pairs_from_covisibility
from pixsfm.refine_hloc import PixSfM

images = Path('data/images')
input_model = Path('data/input_model')
outputs = Path('data/results')

sfm_pairs = outputs / 'pairs-sfm.txt'
features = outputs / 'features.h5'
matches = outputs / 'matches.h5'
sfm_dir = outputs / "sfm"

feature_conf = extract_features.confs['superpoint_aachen']
matcher_conf = match_features.confs['superglue']

references = [str(p.relative_to(images))for p in images.iterdir()]

extract_features.main(feature_conf, images,
                      image_list=references,
                      feature_path=features)
pairs_from_covisibility.main(input_model,
                             sfm_pairs,
                             num_matched=5)
match_features.main(matcher_conf,
                    sfm_pairs,
                    features=features,
                    matches=matches)

refiner = PixSfM(conf="/dependencies/pixel-perfect-sfm/pixsfm/configs/low_memory.yaml")
model, _ = refiner.triangulation(output_dir=sfm_dir,
                                 reference_model_path=input_model,
                                 image_dir=images,
                                 pairs_path=sfm_pairs,
                                 features_path=features,
                                 matches_path=matches
                                 )

and here's the output I got:

[2022/01/27 09:56:05 hloc INFO] Extracting local features with configuration:
{'model': {'max_keypoints': 4096, 'name': 'superpoint', 'nms_radius': 3},
 'output': 'feats-superpoint-n4096-r1024',
 'preprocessing': {'grayscale': True, 'resize_max': 1024}}
[2022/01/27 09:56:05 hloc INFO] Skipping the extraction.
[2022/01/27 09:56:05 hloc INFO] Reading the COLMAP model...
[2022/01/27 09:56:05 hloc INFO] Extracting image pairs from covisibility info...
  0%|                                                                                                                                                                                   | 0/283 [00:00<?, ?it/s][2022/01/27 09:56:05 hloc INFO] Image 28 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 29 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 30 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 57 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 74 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 75 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 131 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 191 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 192 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 202 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 203 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 211 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 223 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 224 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 243 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 244 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 245 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 246 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 247 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 248 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 249 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 264 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 265 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 266 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 269 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 270 does not have any covisibility.
[2022/01/27 09:56:05 hloc INFO] Image 271 does not have any covisibility.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 283/283 [00:00<00:00, 4739.06it/s]
[2022/01/27 09:56:05 hloc INFO] Found 911 pairs.
[2022/01/27 09:56:05 hloc INFO] Matching local features with configuration:
{'model': {'name': 'superglue',
           'sinkhorn_iterations': 50,
           'weights': 'outdoor'},
 'output': 'matches-superglue'}
Loaded SuperGlue model ("outdoor" weights)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 911/911 [00:31<00:00, 28.93it/s]
[2022/01/27 09:56:39 hloc INFO] Finished exporting matches.
[2022/01/27 09:56:40 pixsfm.features.models.s2dnet INFO] Loading S2DNet checkpoint at /dependencies/pixel-perfect-sfm/pixsfm/features/models/checkpoints/s2dnet_weights.pth.
[2022/01/27 09:56:40 pixsfm INFO] Loaded dense extractor with configuration:
{'cache_format': 'chunked',
 'device': 'auto',
 'dtype': 'half',
 'fast_image_load': False,
 'l2_normalize': True,
 'load_cache_on_init': False,
 'max_edge': 1600,
 'model': {'name': 's2dnet', 'num_layers': 1, 'checkpointing': None, 'output_dim': 128, 'pretrained': 's2dnet', 'remove_pooling_layers': False, 'combine': False},
 'overwrite_cache': True,
 'patch_size': 8,
 'pyr_scales': [1.0],
 'resize': 'LANCZOS',
 'sparse': True,
 'use_cache': True}
[2022/01/27 09:56:41 pixsfm INFO] Building matching graph...
[2022/01/27 09:56:41 pixsfm INFO] Extracting dense features...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 256/256 [00:12<00:00, 20.47it/s]
[2022/01/27 09:56:54 pixsfm INFO] Loading featuremaps from H5 File.
100%[████████████████████] 256/256 [00:00, 3605.63it/s]
[2022/01/27 09:56:54 pixsfm INFO] Computing tracks...
[2022/01/27 09:56:54 pixsfm INFO] # graph nodes: 109498
[2022/01/27 09:56:54 pixsfm INFO] # graph edges: 249077
[2022/01/27 09:56:54 pixsfm INFO] # tracks: 34883
[2022/01/27 09:56:54 pixsfm INFO] Start feature-metric keypoint adjustment.
[2022/01/27 09:56:54 pixsfm INFO] Start topological-reference keypoint adjustment.
100%[████████████████████] 109498/109498 [00:11, 9302.35it/s]
[2022/01/27 09:57:06 pixsfm INFO] KA Time: 11.7715s, cost change: 0.0232999 --> 0.0206287
[2022/01/27 09:57:06 hloc INFO] Importing features into the database...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 283/283 [00:00<00:00, 4536.62it/s]
[2022/01/27 09:57:06 hloc INFO] Importing matches into the database...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 911/911 [00:00<00:00, 8098.12it/s]
[2022/01/27 09:57:06 hloc INFO] Performing geometric verification of the matches...
[2022/01/27 09:57:07 hloc INFO] Running 3D triangulation...
[2022/01/27 09:57:10 hloc INFO] Finished the triangulation with statistics:
Reconstruction:
        num_reg_images = 283
        num_cameras = 283
        num_points3D = 1662
        num_observations = 3408
        mean_track_length = 2.05054
        mean_observations_per_image = 12.0424
        mean_reprojection_error = 1.84696
[2022/01/27 09:57:10 pixsfm INFO] Extracting references and costmaps.
Segmentation fault (core dumped)

Did I miss something or is there an actual bug? Might it be related to the lowmemory conf? Is is still ok since I got the log "Finished the triangulation"_ ?

Many thanks !

sarlinpe commented 2 years ago

Thanks for reporting this. The segfault happens during the featuremetric BA but the featuremetric KA and the triangulation seem to succeed. Some pointers:

What are your compute specs? OS, total memory, versions of Python, PyTorch, GCC.
Why does the default configuration crash? Because of OOM? If yes, try to enable the cache without the full low_memory config: PixSfM(conf={"dense_features": {"use_cache": True}})
The reconstructions statistics are quite weak (low track length, few 3D points).
1. Can you run the triangulation with vanilla hloc without PixSfM? hloc.triangulation.main(sfm_dir, input_model, images, sfm_pairs, features, matches)
2. Can you increase the number of pairs in pairs_from_covisibility? e.g. to 20.

ThomasParistech commented 2 years ago

Thanks for your quick reply to my issue

1) I run it on my local DELL-G15 laptop inside a docker.

ARG CUDA_VERSION=11.1
ARG UBUNTU_VERSION=20.04
FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu${UBUNTU_VERSION}

python3.9 gcc-9

torch==1.10.0+cu111 NVIDIA-SMI 470.86

RAM 15GB NVIDIA GeForce RTX 3060 Laptop GPU, 6GB

2) Without the low memory config, the laptop starts freezing since both the 15GB RAM and the 2GB swap are used at 100%.

I used PixSfM(conf={"dense_features": {"use_cache": True}}) as you suggested, and it worked well! The memory consumption reached a plateau instead of keeping to linearly increase like before.

Here's the resulting output

[2022/01/27 13:34:45 hloc INFO] Finished the triangulation with statistics:
Reconstruction:
        num_reg_images = 305
        num_cameras = 305
        num_points3D = 1565
        num_observations = 3351
        mean_track_length = 2.14121
        mean_observations_per_image = 10.9869
        mean_reprojection_error = 1.82589
[2022/01/27 13:34:45 pixsfm INFO] Loading patches from H5 File.
100%[████████████████████] 3351/3351 [00:00, 74466.6it/s]
[2022/01/27 13:34:45 pixsfm INFO] Extracting references.
100%[████████████████████] 1565/1565 [00:00, 7634.14it/s]
[2022/01/27 13:34:45 pixsfm INFO] Reference Extraction Time: 0.205549s
[2022/01/27 13:34:45 pixsfm INFO] Start feature-reference bundle adjustment.
100%[████████████████████] 101/101 [00:03, 25.2689it/s]
[2022/01/27 13:34:49 pixsfm INFO] BA Time: 3.9979s, cost change: 0.0187013 --> 0.0140651

A warning about the use of the low memory conf could be added in the README.

3) Thanks for your tips, I'll try them

sarlinpe commented 2 years ago

Are you able to share the images and the empty COLMAP model? We likely have a bug somewhere in the costmap computation (used by low_memory), having the data would help us fix this.

ThomasParistech commented 2 years ago

Of course Let me quickly record another small scene that I can share with you. What's the best way to share the data with you? I can also provide the docker file

sarlinpe commented 2 years ago

Great. We will need the data that the code crashed with (images + dummy COLMAP model), the exact script you ran and the docker image. If possible zip this together and share a link (e.g. google drive) via email (the address is on my website). Thanks!

cvg / pixel-perfect-sfm

Segfault during triangulation #17