"all input arrays must have the same shape" error while running loFTR pipeline

Hello. Firstly congrats on this great work and thanks for sharing the code.

I have an indoor picture dataset consisting of 247 images each 9248x6936 pixels and taken with the same camera with the same settings. I am trying to run a pipeline based on your Aachen_v1_1 pipeline using loFTR in order to increase the completeness of my point cloud (total number of 3d points). However, I keep getting the error given below in various different configurations while some configurations works smoothly. For example: 1) When I use "loftr" as dense matcher with "netvlad" as feature extractor, I get the same array error below when I use "indoor" weights but it works fine when I set the weights as "outdoor" although it is an indoor picture set. 2) When I use "loftr_superpoint" as dense matcher with "superpointmax as feature extractor, I get the same array error below if I set both "max_error" and "cell_size" to "1" or "2". However if I leave them both as "4" it works fine. 3) When I use "loftr_superpoint" as dense matcher with "superpoint_max" as feature extractor, setting both "max_error" and "cell_size" to 4 but changing "Number of image pairs for SfM" and "Number of image pairs for loc" to 40 and 100, respectively gives me the same array error. Or just changing "Number of image pairs for SfM" from 20 to 40 causing the same error.

Since changes in different settings causing the same error, I cannot pinpoint what the actual reason can be. The only pattern I can see, whenever I change the settings in a way to increase the accuracy and completeness I start to get the same error.

The error is as follows:

[2022/12/01 23:18:34 hloc INFO] Importing matches into the database... 5%| | 250/4893 [00:00<00:02, 1578.48it/s] Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 45, in triangulation.main( File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 224, in main import_matches(image_ids, database, pairs, matches, File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 90, in import_matches matches, scores = get_matches(matches_path, name0, name1) File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 69, in get_matches matches = np.stack([idx, matches[idx]], -1) File "<__array_function__ internals>", line 180, in stack File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 426, in stack raise ValueError('all input arrays must have the same shape') ValueError: all input arrays must have the same shape

Your help is much appreciated.

Can you replace https://github.com/cvg/Hierarchical-Localization/blob/7e6551d78cbb8898520832851614a3438cb2cde7/hloc/utils/io.py#L69 with

    try:
        matches = np.stack([idx, matches[idx]], -1)
    except ValueError as e:
        print(name0, name1)
        print(idx.shape, matches[idx].shape, matches.shape)
        raise e

so we can better understand what is happening here?

cc @Phil26AT in case you have any idea. Maybe related to https://github.com/cvg/Hierarchical-Localization/pull/241.

Thanks a lot for the prompt response. Below is the summary of the first configuration I mentioned above and output after modifying io.py as suggested. I will be running second and the third configs mentioned above and updating shortly.

1) (netvlad + loftr) /pipeline.py --Number of image pairs for SfM 20 --Number of image pairs for loc 50 --max_kps 8192 /extract_features.py --resize_max 512 --interpolation pil_linear /match_dense.py --weights outdoor(*) --resize_max 512 --max_error 1 --cell_size 1

(*): When I change weights to 'indoor', it runs with no error in this config case.

python3 -m hloc.pipelines.NEWbedroom02.pipeline
Configs for dense feature matchers:
{'loftr': {'cell_size': 1,
           'max_error': 1,
           'model': {'name': 'loftr', 'weights': 'outdoor'},
           'output': 'matches-loftr',
           'preprocessing': {'dfactor': 8,
                             'grayscale': True,
                             'resize_max': 1024}},
 'loftr_aachen': {'cell_size': 8,
                  'max_error': 2,
                  'model': {'name': 'loftr', 'weights': 'outdoor'},
                  'output': 'matches-loftr_aachen',
                  'preprocessing': {'dfactor': 8,
                                    'grayscale': True,
                                    'resize_max': 1024}},
 'loftr_netvlad_indoor': {'cell_size': 1,
                          'max_error': 1,
                          'model': {'name': 'loftr', 'weights': 'indoor'},
                          'output': 'matches-loftr_NEWbedroom02',
                          'preprocessing': {'dfactor': 8,
                                            'grayscale': True,
                                            'resize_max': 512}},
 'loftr_superpoint': {'cell_size': 4,
                      'max_error': 4,
                      'model': {'name': 'loftr', 'weights': 'outdoor'},
                      'output': 'matches-loftr_aachen',
                      'preprocessing': {'dfactor': 8,
                                        'grayscale': True,
                                        'resize_max': 1024}},
 'loftr_superpoint_indoor': {'cell_size': 8,
                             'max_error': 4,
                             'model': {'name': 'loftr', 'weights': 'outdoor'},
                             'output': 'matches-loftr_NEWbedroom02',
                             'preprocessing': {'dfactor': 8,
                                               'grayscale': True,
                                               'resize_max': 2048}}}
[2022/12/07 03:06:36 hloc INFO] Reading the COLMAP model...
[2022/12/07 03:06:39 hloc INFO] Extracting image pairs from covisibility info...
100%|                                        | 247/247 [00:01<00:00, 151.08it/s]
[2022/12/07 03:06:40 hloc INFO] Found 4893 pairs.
[2022/12/07 03:06:40 hloc INFO] Extracting semi-dense features with configuration:
{'cell_size': 1,
 'max_error': 1,
 'model': {'name': 'loftr', 'weights': 'indoor'},
 'output': 'matches-loftr_NEWbedroom02',
 'preprocessing': {'dfactor': 8, 'grayscale': True, 'resize_max': 512}}
/home/paperspace/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
[2022/12/07 03:06:44 hloc INFO] Performing dense matching...
100%|                                       | 3441/3441 [05:47<00:00,  9.91it/s]
[2022/12/07 03:12:32 hloc INFO] Assigning matches...
[2022/12/07 03:12:32 hloc INFO] Aggregating keypoints for 247 images.
100%|                                       | 3441/3441 [00:35<00:00, 97.49it/s]
[2022/12/07 03:13:07 hloc INFO] Finished assignment, found 7081.9 keypoints/image (avg.), total 1749234.
[2022/12/07 03:13:07 hloc INFO] Reassign matches with max_error=1.
100%|                                      | 3441/3441 [00:23<00:00, 144.45it/s]
[2022/12/07 03:13:31 hloc INFO] Importing features into the database...
100%|                                       | 247/247 [00:00<00:00, 1713.56it/s]
[2022/12/07 03:13:31 hloc INFO] Importing matches into the database...
  3%| \u258f                                    | 157/4893 [00:00<00:03, 1562.41it/s]20210115_111122.jpg 20210115_111211.jpg
(0,) (0, 2) (0, 2)
  5%| \u2589                                    | 250/4893 [00:00<00:02, 1571.75it/s]
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 45, in <module>
    triangulation.main(
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 224, in main
    import_matches(image_ids, database, pairs, matches,
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 90, in import_matches
    matches, scores = get_matches(matches_path, name0, name1)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 74, in get_matches
    raise e
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 70, in get_matches
    matches = np.stack([idx, matches[idx]], -1)
  File "<__array_function__ internals>", line 180, in stack
  File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 426, in stack
    raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

(superpoint+ loftr) /pipeline.py --Number of image pairs for SfM 20 --Number of image pairs for loc 50 --max_kps 8192 /extract_features.py --resize_max 1024 --interpolation pil_linear /match_dense.py --weights outdoor --resize_max 1024 --max_error 2 ( ) --cell_size 2 ( ) ( * ): When I set max_error and cell_size 2 and 2 or 1 and 1 respectively it fails with the same error but on a different image of the set. If I set them 4 and 4 it runs with no error in this config case.

python3 -m hloc.pipelines.NEWbedroom02.pipeline
Configs for dense feature matchers:
{'loftr': {'cell_size': 1,
           'max_error': 1,
           'model': {'name': 'loftr', 'weights': 'outdoor'},
           'output': 'matches-loftr',
           'preprocessing': {'dfactor': 8,
                             'grayscale': True,
                             'resize_max': 1024}},
 'loftr_aachen': {'cell_size': 8,
                  'max_error': 2,
                  'model': {'name': 'loftr', 'weights': 'outdoor'},
                  'output': 'matches-loftr_aachen',
                  'preprocessing': {'dfactor': 8,
                                    'grayscale': True,
                                    'resize_max': 1024}},
 'loftr_netvlad_indoor': {'cell_size': 1,
                          'max_error': 1,
                          'model': {'name': 'loftr', 'weights': 'indoor'},
                          'output': 'matches-loftr_NEWbedroom02',
                          'preprocessing': {'dfactor': 8,
                                            'grayscale': True,
                                            'resize_max': 512}},
 'loftr_superpoint': {'cell_size': 4,
                      'max_error': 4,
                      'model': {'name': 'loftr', 'weights': 'outdoor'},
                      'output': 'matches-loftr_aachen',
                      'preprocessing': {'dfactor': 8,
                                        'grayscale': True,
                                        'resize_max': 1024}},
 'loftr_superpoint_indoor': {'cell_size': 2,
                             'max_error': 2,
                             'model': {'name': 'loftr', 'weights': 'outdoor'},
                             'output': 'matches-loftr_NEWbedroom02',
                             'preprocessing': {'dfactor': 8,
                                               'grayscale': True,
                                               'resize_max': 1024}}}
[2022/12/07 03:34:21 hloc INFO] Reading the COLMAP model...
[2022/12/07 03:34:24 hloc INFO] Extracting image pairs from covisibility info...
100%|                                                                                                                                                                                                                                                  | 247/247 [00:02<00:00, 121.82it/s]
[2022/12/07 03:34:26 hloc INFO] Found 4893 pairs.
[2022/12/07 03:34:26 hloc INFO] Extracting semi-dense features with configuration:
{'cell_size': 2,
 'max_error': 2,
 'model': {'name': 'loftr', 'weights': 'outdoor'},
 'output': 'matches-loftr_NEWbedroom02',
 'preprocessing': {'dfactor': 8, 'grayscale': True, 'resize_max': 1024}}
/home/paperspace/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
[2022/12/07 03:34:28 hloc INFO] Performing dense matching...
100%|                                                                                                                                                                                                                                                 | 3441/3441 [08:07<00:00,  7.06it/s]
[2022/12/07 03:42:35 hloc INFO] Assigning matches...
[2022/12/07 03:42:35 hloc INFO] Aggregating keypoints for 247 images.
100%|                                                                                                                                                                                                                                                 | 3441/3441 [01:20<00:00, 42.66it/s]
[2022/12/07 03:43:56 hloc INFO] Finished assignment, found 7796.8 keypoints/image (avg.), total 1925816.
[2022/12/07 03:43:56 hloc INFO] Reassign matches with max_error=2.
100%|                                                                                                                                                                                                                                                | 3441/3441 [00:25<00:00, 133.21it/s]
[2022/12/07 03:44:22 hloc INFO] Importing features into the database...
100%|                                                                                                                                                                                                                                                 | 247/247 [00:00<00:00, 1698.01it/s]
[2022/12/07 03:44:22 hloc INFO] Importing matches into the database...
  0%|                                                                                                                                                                                                                                                            | 0/4893 [00:00<?, ?it/s]20210115_111053.jpg 20210115_112155.jpg
(0,) (0, 2) (0, 2)
  2%|    \u258b                                                                                                                                                                                                                                            | 95/4893 [00:00<00:03, 1537.08it/s]
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 45, in <module>
    triangulation.main(
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 224, in main
    import_matches(image_ids, database, pairs, matches,
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 90, in import_matches
    matches, scores = get_matches(matches_path, name0, name1)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 74, in get_matches
    raise e
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 70, in get_matches
    matches = np.stack([idx, matches[idx]], -1)
  File "<__array_function__ internals>", line 180, in stack
  File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 426, in stack
    raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

3) (superpoint+ loftr) /pipeline.py --Number of image pairs for SfM 40 ( ) --Number of image pairs for loc 50 --max_kps 8192 /extract_features.py --resize_max 1024 --interpolation pil_linear /match_dense.py --weights outdoor --resize_max 1024 --max_error 4 --cell_size 4 ( ): When I set "Number of image pairs for SfM" to 40 it fails as below yet at another image. With 20 it runs normally. Also as another combination, I tried to keep "Number of image pairs for SfM" as 40 and increased "resize_max" from 1024 to 2048. This time it ran normally,too. I hope you can spot the problem and advise for a workaround. Thanks in advance.

python3 -m hloc.pipelines.NEWbedroom02.pipeline
Configs for dense feature matchers:
{'loftr': {'cell_size': 1,
           'max_error': 1,
           'model': {'name': 'loftr', 'weights': 'outdoor'},
           'output': 'matches-loftr',
           'preprocessing': {'dfactor': 8,
                             'grayscale': True,
                             'resize_max': 1024}},
 'loftr_aachen': {'cell_size': 8,
                  'max_error': 2,
                  'model': {'name': 'loftr', 'weights': 'outdoor'},
                  'output': 'matches-loftr_aachen',
                  'preprocessing': {'dfactor': 8,
                                    'grayscale': True,
                                    'resize_max': 1024}},
 'loftr_netvlad_indoor': {'cell_size': 1,
                          'max_error': 1,
                          'model': {'name': 'loftr', 'weights': 'indoor'},
                          'output': 'matches-loftr_NEWbedroom02',
                          'preprocessing': {'dfactor': 8,
                                            'grayscale': True,
                                            'resize_max': 512}},
 'loftr_superpoint': {'cell_size': 4,
                      'max_error': 4,
                      'model': {'name': 'loftr', 'weights': 'outdoor'},
                      'output': 'matches-loftr_aachen',
                      'preprocessing': {'dfactor': 8,
                                        'grayscale': True,
                                        'resize_max': 1024}},
 'loftr_superpoint_indoor': {'cell_size': 4,
                             'max_error': 4,
                             'model': {'name': 'loftr', 'weights': 'outdoor'},
                             'output': 'matches-loftr_NEWbedroom02',
                             'preprocessing': {'dfactor': 8,
                                               'grayscale': True,
                                               'resize_max': 1024}}}
[2022/12/07 03:58:54 hloc INFO] Reading the COLMAP model...
[2022/12/07 03:58:57 hloc INFO] Extracting image pairs from covisibility info...
100%|                                                                                                                                                                                                                                                  | 247/247 [00:01<00:00, 144.53it/s]
[2022/12/07 03:58:58 hloc INFO] Found 9344 pairs.
[2022/12/07 03:58:58 hloc INFO] Extracting semi-dense features with configuration:
{'cell_size': 4,
 'max_error': 4,
 'model': {'name': 'loftr', 'weights': 'outdoor'},
 'output': 'matches-loftr_NEWbedroom02',
 'preprocessing': {'dfactor': 8, 'grayscale': True, 'resize_max': 1024}}
/home/paperspace/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
[2022/12/07 03:59:00 hloc INFO] Performing dense matching...
100%|                                                                                                                                                                                                                                                 | 6014/6014 [13:58<00:00,  7.17it/s]
[2022/12/07 04:12:59 hloc INFO] Assigning matches...
[2022/12/07 04:12:59 hloc INFO] Aggregating keypoints for 247 images.
100%|                                                                                                                                                                                                                                                 | 6014/6014 [01:58<00:00, 50.75it/s]
[2022/12/07 04:14:57 hloc INFO] Finished assignment, found 7913.6 keypoints/image (avg.), total 1954651.
[2022/12/07 04:14:57 hloc INFO] Reassign matches with max_error=4.
100%|                                                                                                                                                                                                                                                | 6014/6014 [00:41<00:00, 145.82it/s]
[2022/12/07 04:15:39 hloc INFO] Importing features into the database...
100%|                                                                                                                                                                                                                                                 | 247/247 [00:00<00:00, 1653.35it/s]
[2022/12/07 04:15:39 hloc INFO] Importing matches into the database...
 14%|                                  \u258d                                                                                                                                                                                                            | 1345/9344 [00:00<00:04, 1724.69it/s]20210115_111302.jpg 20210115_112632.jpg
(0,) (0, 2) (0, 2)
 16%|                                     \u258a                                                                                                                                                                                                         | 1477/9344 [00:00<00:04, 1684.43it/s]
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 45, in <module>
    triangulation.main(
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 224, in main
    import_matches(image_ids, database, pairs, matches,
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 90, in import_matches
    matches, scores = get_matches(matches_path, name0, name1)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 74, in get_matches
    raise e
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 70, in get_matches
    matches = np.stack([idx, matches[idx]], -1)
  File "<__array_function__ internals>", line 180, in stack
  File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 426, in stack
    raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

Thanks for the detailed logs. Can you please try PR https://github.com/cvg/Hierarchical-Localization/pull/254? Please remove existing match and features files beforehand.

Hello,

I ran the first configuration again with "indoor" weights which was the failed case. It passed the previous stage. However, it failed again with a different type of array error at a different stage. Copying the output below.

Also I noticed that -as a side question- indoor weight produced far more less (only 1855) 3D points, while using outdoor weights produces 34013 points during 3D reconstruction. The image set belongs to an indoor area (a simple bedroom), so I would expect the number to be higher when "indoor" weights used. Is this an expected behavior?

python3 -m hloc.pipelines.NEWbedroom02.pipeline
Configs for dense feature matchers:
{'loftr': {'cell_size': 1,
           'max_error': 1,
           'model': {'name': 'loftr', 'weights': 'outdoor'},
           'output': 'matches-loftr',
           'preprocessing': {'dfactor': 8,
                             'grayscale': True,
                             'resize_max': 1024}},
 'loftr_aachen': {'cell_size': 8,
                  'max_error': 2,
                  'model': {'name': 'loftr', 'weights': 'outdoor'},
                  'output': 'matches-loftr_aachen',
                  'preprocessing': {'dfactor': 8,
                                    'grayscale': True,
                                    'resize_max': 1024}},
 'loftr_netvlad_indoor': {'cell_size': 1,
                          'max_error': 1,
                          'model': {'name': 'loftr', 'weights': 'indoor'},
                          'output': 'matches-loftr_NEWbedroom02',
                          'preprocessing': {'dfactor': 8,
                                            'grayscale': True,
                                            'resize_max': 512}},
 'loftr_superpoint': {'cell_size': 4,
                      'max_error': 4,
                      'model': {'name': 'loftr', 'weights': 'outdoor'},
                      'output': 'matches-loftr_aachen',
                      'preprocessing': {'dfactor': 8,
                                        'grayscale': True,
                                        'resize_max': 1024}},
 'loftr_superpoint_indoor': {'cell_size': 4,
                             'max_error': 4,
                             'model': {'name': 'loftr', 'weights': 'outdoor'},
                             'output': 'matches-loftr_NEWbedroom02',
                             'preprocessing': {'dfactor': 8,
                                               'grayscale': True,
                                               'resize_max': 2560}}}
[2022/12/08 00:40:27 hloc INFO] Reading the COLMAP model...
[2022/12/08 00:40:30 hloc INFO] Extracting image pairs from covisibility info...
100%|                                                                                                                                                                                                                                                  | 247/247 [00:01<00:00, 124.49it/s]
[2022/12/08 00:40:32 hloc INFO] Found 4893 pairs.
[2022/12/08 00:40:32 hloc INFO] Extracting semi-dense features with configuration:
{'cell_size': 1,
 'max_error': 1,
 'model': {'name': 'loftr', 'weights': 'indoor'},
 'output': 'matches-loftr_NEWbedroom02',
 'preprocessing': {'dfactor': 8, 'grayscale': True, 'resize_max': 512}}
/home/paperspace/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
[2022/12/08 00:40:34 hloc INFO] Performing dense matching...
100%|                                                                                                                                                                                                                                                 | 3441/3441 [06:25<00:00,  8.93it/s]
[2022/12/08 00:46:59 hloc INFO] Assigning matches...
[2022/12/08 00:46:59 hloc INFO] Aggregating keypoints for 247 images.
100%|                                                                                                                                                                                                                                                 | 3441/3441 [00:45<00:00, 76.01it/s]
[2022/12/08 00:47:45 hloc INFO] Finished assignment, found 7081.9 keypoints/image (avg.), total 1749234.
[2022/12/08 00:47:45 hloc INFO] Reassign matches with max_error=1.
100%|                                                                                                                                                                                                                                                | 3441/3441 [00:27<00:00, 124.77it/s]
[2022/12/08 00:48:13 hloc INFO] Importing features into the database...
100%|                                                                                                                                                                                                                                                 | 247/247 [00:00<00:00, 1396.66it/s]
[2022/12/08 00:48:13 hloc INFO] Importing matches into the database...
100%|                                                                                                                                                                                                                                               | 4893/4893 [00:02<00:00, 1959.14it/s]
[2022/12/08 00:48:16 hloc INFO] Performing geometric verification of the matches...
100%|                                                                                                                                                                                                                                                   | 247/247 [01:14<00:00,  3.31it/s]
[2022/12/08 00:49:30 hloc INFO] mean/med/min/max valid matches 3.93/2.77/0.00/33.15%.
[2022/12/08 00:49:30 hloc INFO] Running 3D triangulation...
[2022/12/08 00:51:02 hloc INFO] Finished the triangulation with statistics:
Reconstruction:
    num_reg_images = 247
    num_cameras = 1
    num_points3D = 1855
    num_observations = 4806
    mean_track_length = 2.59084
    mean_observations_per_image = 19.4575
    mean_reprojection_error = 1.44981
[2022/12/08 00:51:03 hloc INFO] Extracting local features with configuration:
{'model': {'name': 'netvlad'},
 'output': 'global-feats-netvlad',
 'preprocessing': {'resize_max': 512}}
[2022/12/08 00:51:03 hloc INFO] Found 247 images in root datasets/NEWbedroom02/images/images_upright.
100%|                                                                                                                                                                                                                                                   | 247/247 [07:07<00:00,  1.73s/it]
[2022/12/08 00:58:16 hloc INFO] Finished exporting features.
[2022/12/08 00:58:16 hloc INFO] Extracting image pairs from a retrieval database.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 54, in <module>
    pairs_from_retrieval.main(
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pairs_from_retrieval.py", line 94, in main
    query_desc = get_descriptors(query_names, descriptors)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pairs_from_retrieval.py", line 42, in get_descriptors
    return torch.from_numpy(np.stack(desc, 0)).float()
  File "<__array_function__ internals>", line 180, in stack
  File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 422, in stack
    raise ValueError('need at least one array to stack')
ValueError: need at least one array to stack

I recommend to always use outdoor weights, regardless of the type of environment. The indoor model likely overfits to the camera statistics of the ScanNet dataset.

Regarding your error: what are you trying to achieve? do you have query images to localize? If not, all steps after the triangulation/reconstruction are irrelevant and should be discarded from the original pipeline. This simply fails because it cannot find query images with the prefix query.

Thanks for the explanation. I have gone through it and understood it better. I have also run the other 2 config cases and they are working fine. I believe this issue can be closed.

I have 2 off-topic questions if you don't mind answering, though. 1) Given the photos taken by the same (consumer grade) camera under same lighting conditions, I am trying to maximize accuracy and completeness of indoor scenes. Going through the literature I think loftr+superpoint-->hloc-->pixelperfect is my best option for the time being. For hloc, assuming unlimited compute resources or/and time to compute, what would be the other parameters to tweak to achieve this? For example would changing 'dfactor' value or setting 'grayscale' false help? If you have a guideline out somewhere that I am missing, I appreciate if you could guide me to that. 2) Authors of LoFTR released another repo named QuadTreeAttention* with better performance** than LoFTR. Any plans to implement that into hloc?

( * ) https://github.com/Tangshitao/QuadTreeAttention ( ** ) https://www.kaggle.com/competitions/image-matching-challenge-2022/discussion/328805

Thanks for confirming that everything works.

Grayscale is a requirement of the feature extractor and cannot be tweaked. Increasing the size of the input images and matching all points (not only SuperPoint keypoints) can both increase the density of the 3D point cloud.
We do not have plan to integrate this in the near future.

cvg / Hierarchical-Localization

"all input arrays must have the same shape" error while running loFTR pipeline #253