Different mathcing results on two machines

[ccc14023748]

I'm using OpenSfM on different machines, one is a workstation with Ubuntu and the other is a server with Oracle Linux. After running a set of 176 images, I get successful results on the workstation, but always get failed matching results on the server (0 robust matches for all images and consequently 0 tracks and 0 reconstructions).

Successful results on the workstation: (e.g.) 2021-09-03 12:44:19,093 DEBUG: Matching G0030109.JPG and G0030114.JPG. Matcher: FLANN (symmetric) T-desc: 0.547 T-robust: 0.014 T-total: 0.561 Matches: 139 Robust: 76 Success: True

Failed results on the server: (e.g.) 2021-09-02 23:43:34,546 DEBUG: Matching G0030109.JPG and G0030114.JPG. Matcher: FLANN (symmetric) T-desc: 1.122 T-robust: 0.492 T-total: 1.614 Matches: 132 Robust: 0 Success: False

I'm using exactly same config.yaml and I've confirmed the data is fine since I got successful results on the workstation and other SfM softwares (e.g. COLMAP). Also, I've confirmed the extracted SIFT features are same (exactly same size of features.npz) on both machines.

What would be the causes of the inconsistent matching results?

1. Different version of Python dependencies (e.g. Numpy, opencv-python) on different machines?
2. Different version of OpenSfM? It was installed on the server by our lab administrator, but I've tested older and newer versions on the workstation and all got successful results.
3. Any other possible reason?

[YanNoun]

Hi @ccc14023748,

Differences in 2D matching can be due to FLANN (see here https://stackoverflow.com/questions/40005790/flannbasedmatcher-gives-different-results-when-run-on-different-computers for a similar issue), as by default OpenSfM uses several KMeans trees that will randomized (https://github.com/mapillary/OpenSfM/blob/main/opensfm/config.py#L54).

Setting flann_tree to 1 instead could make FLANN more deterministic, but then you will have deterministic issues downstream in opensfm reconstruct due to ceres pojnter-based ordering and schur-complement multithreaded construction.

As a note, OpenSfM RANSACs use fixed random seed : https://github.com/mapillary/OpenSfM/blob/main/opensfm/src/robust/random_sampler.h#L10

Let me know if that answer you question.

Yann

[ccc14023748]

Thank you so much for the detailed explanation. I've change the matcher type in config.yaml, however, still got different results(robust matches) on different machines. Actually, what I need is to successfully run SfM on the server, no matter what kind of matching configurations. Is there any way to achieve that?

in config.yaml: matcher_type: BRUTEFORCE

Successful results on the workstation: 2021-09-26 16:09:03,000 DEBUG: Matching G0030114.JPG and G0030109.JPG. Matcher: BRUTEFORCE (symmetric) T-desc: 72.846 T-robust: 0.008 T-total: 72.854 Matches: 153 Robust: 109 Success: True

Failed results on the server: 2021-09-26 15:41:37,090 DEBUG: Matching G0030109.JPG and G0030114.JPG. Matcher: BRUTEFORCE (symmetric) T-desc: 41.407 T-robust: 0.514 T-total: 41.921 Matches: 153 Robust: 0 Success: False

[YanNoun]

Hi @ccc14023748,

Thanks for reporting the issue, this is super interesting. Could you run OpenSfM unit tests on the server with :

python3 -m pytest && cd cmake_build && ctest && cd -

Thank you,

Yann

[ccc14023748]

We found that when running tests on test*.py, some of them threw this error Segmentation fault (core dumped). It is from opensfm import multiview, types causing the problem, although we don't know why. And we accidentally found a strange solution, which is adding from opensfm import commands before importing above-mentioned modules in the test*.py that threw errors. Finally, the pytest and ctest results are as follows:

============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /opt/opensfm-0.5.1-python-3.7.10-cpu, configfile: setup.cfg, testpaths: opensfm
plugins: typeguard-2.12.1, anyio-3.3.1, hydra-core-1.1.1
collected 235 items

opensfm/geo.py ....                                                      [  1%]
opensfm/multiview.py .........                                           [  5%]
opensfm/transformations.py ..........................................    [ 23%]
opensfm/upright.py .                                                     [ 23%]
opensfm/synthetic_data/synthetic_scene.py .                              [ 24%]
opensfm/test/test_bundle.py ...............                              [ 30%]
opensfm/test/test_commands.py F                                          [ 31%]
opensfm/test/test_dataset.py .                                           [ 31%]
opensfm/test/test_datastructures.py .................................... [ 46%]
.......................                                                  [ 56%]
opensfm/test/test_dense.py ..                                            [ 57%]
opensfm/test/test_geo.py .....                                           [ 59%]
opensfm/test/test_geometry.py ..                                         [ 60%]
opensfm/test/test_io.py ........                                         [ 63%]
opensfm/test/test_matching.py ...F.F                                     [ 66%]
opensfm/test/test_multiview.py ......F..                                 [ 70%]
opensfm/test/test_pairs_selection.py .....                               [ 72%]
opensfm/test/test_reconstruction_alignment.py ........                   [ 75%]
opensfm/test/test_reconstruction_incremental.py FF                       [ 76%]
opensfm/test/test_reconstruction_resect.py ..                            [ 77%]
opensfm/test/test_reconstruction_shot_neighborhood.py ......             [ 80%]
opensfm/test/test_rig.py ..                                              [ 80%]
opensfm/test/test_robust.py ........F...                                 [ 85%]
opensfm/test/test_stats.py ..............                                [ 91%]
opensfm/test/test_triangulation.py ...F.                                 [ 94%]
opensfm/test/test_types.py .........                                     [ 97%]
opensfm/test/test_undistort.py .                                         [ 98%]
opensfm/test/test_vlad.py ...                                            [ 99%]
opensfm/test/large/test_tools.py .                                       [100%]

=================================== FAILURES ===================================
_________________________________ test_run_all _________________________________

tmpdir = local('/tmp/pytest-of-ccc14023748/pytest-3/test_run_all0')

    def test_run_all(tmpdir):
        data = data_generation.create_berlin_test_folder(tmpdir)
        run_all_commands = [
            commands.extract_metadata,
            commands.detect_features,
            commands.match_features,
            commands.create_tracks,
            commands.reconstruct,
            commands.bundle,
            commands.reconstruct_from_prior,
            commands.mesh,
            commands.undistort,
            commands.compute_depthmaps,
            commands.export_ply,
            commands.export_visualsfm,
            commands.export_openmvs,
            commands.export_pmvs,
            commands.export_bundler,
            commands.export_colmap,
            commands.compute_statistics,
            commands.export_report,
        ]

        output_rec_path = join(data.data_path, "rec_prior.json")
        command_options = {
            commands.reconstruct_from_prior: [
                "--input",
                join(data.data_path, "reconstruction.json"),
                "--output",
                output_rec_path,
            ]
        }

        for module in run_all_commands:
            command = module.Command()
            options = command_options.get(module, [])
>           run_command(command, [data.data_path] + options)

opensfm/test/test_commands.py:51: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
opensfm/test/test_commands.py:12: in run_command
    command.run(dataset.DataSet(parsed_args.dataset), parsed_args)
opensfm/commands/command.py:12: in run
    self.run_impl(data, args)
opensfm/commands/reconstruct_from_prior.py:11: in run_impl
    reconstruct_from_prior.run_dataset(dataset, args.input, args.output)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = <opensfm.dataset.DataSet object at 0x7f15cfd08850>
input = '/tmp/pytest-of-ccc14023748/pytest-3/test_run_all0/berlin/reconstruction.json'
output = '/tmp/pytest-of-ccc14023748/pytest-3/test_run_all0/berlin/rec_prior.json'

    def run_dataset(data: DataSetBase, input: str, output: str):
        """ Reconstruct the from a prior reconstruction. """

        tracks_manager = data.load_tracks_manager()
        rec_prior = data.load_reconstruction(input)
        if len(rec_prior) > 0:
            report, rec = reconstruction.reconstruct_from_prior(
                data, tracks_manager, rec_prior[0]
            )
        # pyre-fixme[61]: `rec` may not be initialized here.
>       data.save_reconstruction([rec], output)
E       UnboundLocalError: local variable 'rec' referenced before assignment

opensfm/actions/reconstruct_from_prior.py:15: UnboundLocalError
______________________________ test_match_images _______________________________

scene_synthetic = <opensfm.synthetic_data.synthetic_scene.SyntheticInputData object at 0x7f15cff71a90>

    def test_match_images(scene_synthetic):
        reference = scene_synthetic.reconstruction
        synthetic = synthetic_dataset.SyntheticDataSet(
            reference,
            scene_synthetic.exifs,
            scene_synthetic.features,
            scene_synthetic.tracks_manager,
        )

        synthetic.matches_exists = lambda im: False
        synthetic.save_matches = lambda im, m: False

        override = {}
        override["matching_gps_neighbors"] = 0
        override["matching_gps_distance"] = 0
        override["matching_time_neighbors"] = 2

        images = sorted(synthetic.images())
        pairs, _ = matching.match_images(synthetic, override, images, images)
        matching.save_matches(synthetic, images, pairs)

        for i in range(len(images) - 1):
            pair = images[i], images[i + 1]
            matches = pairs.get(pair)
            if matches is None or len(matches) == 1:
                matches = pairs.get(pair[::-1])
>           assert len(matches) > 25
E           assert 0 > 25
E            +  where 0 = len(array([], dtype=float64))

opensfm/test/test_matching.py:105: AssertionError
__________________________ test_triangulation_inliers __________________________

pairs_and_their_E = [(array([[-0.26614, -0.05946,  1.02883],
       [ 0.09372,  0.10866,  1.08626],
       [ 0.08659, -0.15307,  1.02631],...-0.03103,  0.04062],
       [ 0.41764, -0.06357, -0.     ]]), <opensfm.pygeometry.Pose object at 0x7f15cfcefaf0>), ...]

    def test_triangulation_inliers(pairs_and_their_E):
        for f1, f2, _, pose in pairs_and_their_E:
            Rt = pose.get_cam_to_world()[:3]

            count_outliers = np.random.randint(0, len(f1) / 10)
            f1[:count_outliers, :] += np.random.uniform(0, 1e-1, size=(count_outliers, 3))

            inliers = matching.compute_inliers_bearings(f1, f2, Rt[:, :3], Rt[:, 3])
>           assert sum(inliers) >= len(f1) - count_outliers
E           assert 0 >= (1000 - 55)
E            +  where 0 = sum([False, False, False, False, False, False, ...])
E            +  and   1000 = len(array([[-0.26614, -0.05946,  1.02883],\n       [ 0.09372,  0.10866,  1.08626],\n       [ 0.08659, -0.15307,  1.02631],\n ... \n       [-0.00842, -0.19968,  0.97983],\n       [-0.08046,  0.20414,  0.97563],\n       [-0.07495, -0.07064,  0.99468]]))

opensfm/test/test_matching.py:129: AssertionError
______________________ test_relative_pose_from_essential _______________________

pairs_and_their_E = [(array([[-0.28027, -0.14164,  0.94941],
       [-0.00353,  0.09363,  0.9956 ],
       [ 0.05248, -0.19468,  0.97946],...-0.03103,  0.04062],
       [ 0.41764, -0.06357, -0.     ]]), <opensfm.pygeometry.Pose object at 0x7f15cfef6170>), ...]

    def test_relative_pose_from_essential(pairs_and_their_E):
        for f1, f2, E, pose in pairs_and_their_E:

            result = pygeometry.relative_pose_from_essential(E, f1, f2)

            pose = copy.deepcopy(pose)
            pose.translation /= np.linalg.norm(pose.translation)

            expected = pose.get_world_to_cam()[:3]
>           assert np.allclose(expected, result, rtol=1e-10)
E           assert False
E            +  where False = <function allclose at 0x7f16103e64d0>(array([[ 0.99035, -0.04131, -0.13231,  0.8713 ],\n       [ 0.03174,  0.99678, -0.07362,  0.48484],\n       [ 0.13492,  0.06871,  0.98847,  0.07592]]), array([[ 0.,  0.,  0.,  0.],\n       [ 0.,  0.,  0.,  0.],\n       [ 0.,  0.,  0.,  0.]]), rtol=1e-10)
E            +    where <function allclose at 0x7f16103e64d0> = np.allclose

opensfm/test/test_multiview.py:113: AssertionError
_______________________ test_reconstruction_incremental ________________________

scene_synthetic = <opensfm.synthetic_data.synthetic_scene.SyntheticInputData object at 0x7f15cfc1d8d0>

    def test_reconstruction_incremental(
        scene_synthetic: synthetic_scene.SyntheticInputData,
    ):
        reference = scene_synthetic.reconstruction
        dataset = synthetic_dataset.SyntheticDataSet(
            reference,
            scene_synthetic.exifs,
            scene_synthetic.features,
            scene_synthetic.tracks_manager,
            scene_synthetic.gcps,
        )

        dataset.config["bundle_compensate_gps_bias"] = True
        dataset.config["bundle_use_gcp"] = True
        _, reconstructed_scene = reconstruction.incremental_reconstruction(
            dataset, scene_synthetic.tracks_manager
        )
        errors = synthetic_scene.compare(
            reference,
            scene_synthetic.gcps,
>           reconstructed_scene[0],
        )
E       IndexError: list index out of range

opensfm/test/test_reconstruction_incremental.py:26: IndexError
_____________________ test_reconstruction_incremental_rig ______________________

scene_synthetic_rig = <opensfm.synthetic_data.synthetic_scene.SyntheticInputData object at 0x7f15c7d89050>

    def test_reconstruction_incremental_rig(
        scene_synthetic_rig: synthetic_scene.SyntheticInputData,
    ):
        reference = scene_synthetic_rig.reconstruction
        dataset = synthetic_dataset.SyntheticDataSet(
            reference,
            scene_synthetic_rig.exifs,
            scene_synthetic_rig.features,
            scene_synthetic_rig.tracks_manager,
        )

        dataset.config["align_method"] = "orientation_prior"
        _, reconstructed_scene = reconstruction.incremental_reconstruction(
            dataset, scene_synthetic_rig.tracks_manager
        )
>       errors = synthetic_scene.compare(reference, {}, reconstructed_scene[0])
E       IndexError: list index out of range

opensfm/test/test_reconstruction_incremental.py:67: IndexError
______________________ test_outliers_relative_pose_ransac ______________________

pairs_and_their_E = [(array([[-0.28027, -0.14164,  0.94941],
       [-0.00353,  0.09363,  0.9956 ],
       [ 0.05248, -0.19468,  0.97946],...-0.03103,  0.04062],
       [ 0.41764, -0.06357, -0.     ]]), <opensfm.pygeometry.Pose object at 0x7f15cfbc8170>), ...]

    def test_outliers_relative_pose_ransac(pairs_and_their_E):
        for f1, f2, _, pose in pairs_and_their_E:
            points = np.concatenate((f1, f2), axis=1)

            scale = 1e-3
            points += np.random.rand(*points.shape) * scale

            ratio_outliers = 0.3
            add_outliers(ratio_outliers, points, 0.1, 1.0)

            f1, f2 = points[:, 0:3], points[:, 3:6]
            f1 /= np.linalg.norm(f1, axis=1)[:, None]
            f2 /= np.linalg.norm(f2, axis=1)[:, None]

            scale_eps_ratio = 1e-1
            params = pyrobust.RobustEstimatorParams()
            params.iterations = 1000
            result = pyrobust.ransac_relative_pose(
                f1, f2, scale * (1.0 + scale_eps_ratio), params, pyrobust.RansacType.RANSAC
            )

            expected = pose.get_world_to_cam()[:3]
            expected[:, 3] /= np.linalg.norm(expected[:, 3])

            tolerance = 0.12
            inliers_count = (1 - ratio_outliers) * len(points)
>           assert np.isclose(len(result.inliers_indices), inliers_count, rtol=tolerance)
E           assert False
E            +  where False = <function isclose at 0x7f16103e6710>(1, 700.0, rtol=0.12)
E            +    where <function isclose at 0x7f16103e6710> = np.isclose
E            +    and   1 = len([44])
E            +      where [44] = <opensfm.pyrobust.ScoreInfoMatrix34d object at 0x7f15c7da64b0>.inliers_indices

opensfm/test/test_robust.py:227: AssertionError
____________________ test_triangulate_two_bearings_midpoint ____________________

    def test_triangulate_two_bearings_midpoint():
        o1 = np.array([0.0, 0, 0])
        b1 = unit_vector([0.0, 0, 1])
        o2 = np.array([1.0, 0, 0])
        b2 = unit_vector([-1.0, 0, 1])
        ok, X = pygeometry.triangulate_two_bearings_midpoint([o1, o2], [b1, b2])
>       assert ok is True
E       assert False is True

opensfm/test/test_triangulation.py:87: AssertionError
=============================== warnings summary ===============================
../python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/cacheprovider.py:428
  /opt/python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/cacheprovider.py:428: PytestCacheWarning: could not create cache path /opt/opensfm-0.5.1-python-3.7.10-cpu/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/cacheprovider.py:382
  /opt/python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/cacheprovider.py:382: PytestCacheWarning: could not create cache path /opt/opensfm-0.5.1-python-3.7.10-cpu/.pytest_cache/v/cache/lastfailed
    config.cache.set("cache/lastfailed", self.lastfailed)

../python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/stepwise.py:49
  /opt/python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/stepwise.py:49: PytestCacheWarning: could not create cache path /opt/opensfm-0.5.1-python-3.7.10-cpu/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED opensfm/test/test_commands.py::test_run_all - UnboundLocalError: local...
FAILED opensfm/test/test_matching.py::test_match_images - assert 0 > 25
FAILED opensfm/test/test_matching.py::test_triangulation_inliers - assert 0 >...
FAILED opensfm/test/test_multiview.py::test_relative_pose_from_essential - as...
FAILED opensfm/test/test_reconstruction_incremental.py::test_reconstruction_incremental
FAILED opensfm/test/test_reconstruction_incremental.py::test_reconstruction_incremental_rig
FAILED opensfm/test/test_robust.py::test_outliers_relative_pose_ransac - asse...
FAILED opensfm/test/test_triangulation.py::test_triangulate_two_bearings_midpoint
============ 8 failed, 227 passed, 3 warnings in 166.88s (0:02:46) =============
Test project /opt/opensfm-0.5.1-python-3.7.10-cpu/cmake_build
Cannot create directory /opt/opensfm-0.5.1-python-3.7.10-cpu/cmake_build/Testing/Temporary
Cannot create log file: LastTest.log
    Start 1: foundation_test
1/7 Test #1: foundation_test ..................   Passed    0.07 sec
    Start 2: bundle_test
2/7 Test #2: bundle_test ......................   Passed    0.08 sec
    Start 3: dense_test
3/7 Test #3: dense_test .......................   Passed    0.06 sec
    Start 4: geo_test
4/7 Test #4: geo_test .........................   Passed    0.06 sec
    Start 5: geometry_test
5/7 Test #5: geometry_test ....................   Passed    0.10 sec
    Start 6: sfm_test
6/7 Test #6: sfm_test .........................   Passed    0.07 sec
    Start 7: map_test
7/7 Test #7: map_test .........................   Passed    0.08 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) =   0.70 sec

[YanNoun]

Hi @ccc14023748,

The failing test test_triangulate_two_bearings_midpoint is highly suspicious as it is a very simple test that just triangulate a point. Under the hood it is really simple code using Eigen, culminating with computing a 2x2 inverse. The most recent change is https://github.com/mapillary/OpenSfM/blob/main/opensfm/src/geometry/triangulation.h#L71 with the #ifdef __aarch64__ so you could try with and without.

python -m pytest -k 'test_triangulate_two_bearings_midpoint' is a good proxy for checking if the issue is solved.

Yann

[ccc14023748]

I've tried hiding line 71-73, 75 as https://github.com/mapillary/OpenSfM/commit/d46847105c28d07393b72f1804e6dcc70bd22f3a#diff-0de25540dd4d0abbddb357d6b3a0cb9500a0730fb1e1219745f0a4896e7d92b5 and ran pytest. However, the result is exactly the same.

============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /opt/opensfm-0.5.1-python-3.7.10-cpu, configfile: setup.cfg, testpaths: opensfm
plugins: typeguard-2.12.1, anyio-3.3.1, hydra-core-1.1.1
collected 236 items / 234 deselected / 2 selected

opensfm/test/test_triangulation.py F.                                    [100%]

=================================== FAILURES ===================================
____________________ test_triangulate_two_bearings_midpoint ____________________

    def test_triangulate_two_bearings_midpoint():
        o1 = np.array([0.0, 0, 0])
        b1 = unit_vector([0.0, 0, 1])
        o2 = np.array([1.0, 0, 0])
        b2 = unit_vector([-1.0, 0, 1])
        ok, X = pygeometry.triangulate_two_bearings_midpoint([o1, o2], [b1, b2])
>       assert ok is True
E       assert False is True

opensfm/test/test_triangulation.py:87: AssertionError
=============================== warnings summary ===============================
../python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/cacheprovider.py:428
  /opt/python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/cacheprovider.py:428: PytestCacheWarning: could not create cache path /opt/opensfm-0.5.1-python-3.7.10-cpu/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/cacheprovider.py:382
  /opt/python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/cacheprovider.py:382: PytestCacheWarning: could not create cache path /opt/opensfm-0.5.1-python-3.7.10-cpu/.pytest_cache/v/cache/lastfailed
    config.cache.set("cache/lastfailed", self.lastfailed)

../python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/stepwise.py:49
  /opt/python-3.7.10-cpu/lib/python3.7/site-packages/_pytest/stepwise.py:49: PytestCacheWarning: could not create cache path /opt/opensfm-0.5.1-python-3.7.10-cpu/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED opensfm/test/test_triangulation.py::test_triangulate_two_bearings_midpoint
=========== 1 failed, 1 passed, 234 deselected, 3 warnings in 14.31s ===========