Reproduce DAVIS dataset with trouble

[lan-creator]

Hello, I'm got in some trouble when I want to reproduce DAVIS dataset using colmap. When I try to use colmap mapper, there are always pro like "No good initial pair found at reconstruction" and the rgbs and poses I got always were only a few or a dozen, and I could not get all poses of frames in one scene e.g. dog scene I only get poses of frame 05, 07, 48, 58. The colmap setting is as below in $PWD/pgdvs/preprocess/colmap_processor.py:

`

def run_colmap(args):

max_num_matches = 132768  # colmap setting
workspace_path = pathlib.Path(args.workspace_path)

workspace_path.mkdir(parents=True, exist_ok=True)
rgb_dir = pathlib.Path(args.image_path)
mask_dir = pathlib.Path(args.mask_path)

exts = PIL.Image.registered_extensions()
supported_exts = {ex for ex, f in exts.items() if f in PIL.Image.OPEN}

rgb_f_list = []
for tmp_ext in supported_exts:
    rgb_f_list = rgb_f_list + list(sorted(rgb_dir.glob(f"*{tmp_ext}")))
for tmp_rgb_f in rgb_f_list:
    tmp_mask_f = mask_dir / f"{tmp_rgb_f.name}.png"
    assert tmp_mask_f.exists(), rgb_f_list

if not (workspace_path / "database.db").exists() or args.overwrite:
    # https://colmap.github.io/faq.html#mask-image-regions
    # Features will only be extracted from areas with mask values of 1

    # fmt: off
    cmd_feat = [
        f"{args.colmap_bin}",
        "feature_extractor",
        "--database_path", f"{args.workspace_path}/database.db",
        "--image_path", f"{args.image_path}",
        "--ImageReader.mask_path", f"{args.mask_path}",
        "--ImageReader.camera_model", "SIMPLE_RADIAL",
        "--ImageReader.single_camera", "1",
        "--SiftExtraction.num_threads", "1",
        "--SiftExtraction.gpu_index", "0",
    ]

    # "--ImageReader.default_focal_length_factor", "0.95",
    # "--SiftExtraction.peak_threshold", "0.004",
    # "--SiftExtraction.max_num_features", "8192",
    # "--SiftExtraction.edge_threshold", "16",

    # fmt: on
    run_cmd(cmd_feat)

    # fmt: off
    cmd_match = [
        f"{args.colmap_bin}",
        "exhaustive_matcher",
        "--database_path", f"{args.workspace_path}/database.db",
        "--SiftMatching.guided_matching", "1",
    ]
    # "--SiftMatching.multiple_models", "1", # 3.10 default true
    # "--SiftMatching.max_ratio", "0.8",
    # "--SiftMatching.max_error", "4.0",
    # "--SiftMatching.max_distance", "0.7",
    # "--SiftMatching.max_num_matches", f"{max_num_matches}",

    # fmt: on
    run_cmd(cmd_match)

if not (workspace_path / "sparse").exists() or args.overwrite:
    (workspace_path / "sparse").mkdir(exist_ok=True, parents=True)

    # fmt: off
    cmd_map = [
        f"{args.colmap_bin}",
        "mapper",
        "--database_path", f"{args.workspace_path}/database.db",
        "--image_path", f"{args.image_path}",
        "--output_path", f"{args.workspace_path}/sparse",
        "--Mapper.abs_pose_min_inlier_ratio", "0.5",
        "--Mapper.abs_pose_min_num_inliers", "50",
        "--Mapper.init_max_forward_motion", "1",
        "--Mapper.ba_local_num_images", "15",
    ]
    # fmt: on
    run_cmd(cmd_map)

if not pathlib.Path(args.undistort_path).exists() or args.overwrite:
    pathlib.Path(args.undistort_path).mkdir(exist_ok=True, parents=True)

    # fmt: off
    cmd_undist = [
        f"{args.colmap_bin}",
        "image_undistorter",
        "--input_path", f"{args.workspace_path}/sparse/0",
        "--image_path", f"{args.image_path}",
        "--output_path", f"{args.undistort_path}",
        "--output_type", "COLMAP",
    ]
    # fmt: on
    run_cmd(cmd_undist)`

The only different code is just in exhaustive_matcher and the cmd "--SiftMatching.multiple_models", "1" is set to default true in colmap 3.10. I've searched https://github.com/colmap/colmap/issues/1490 and try changing init_max_forward_motion, which is not helpful. And for mask I follow the instruction to produce, so I wonder if there are some settings or hyperparameters I'm using incorrectly. Maybe colmap version problem as I use colmap 3.10? But the colmap version seems not to cause such a big problem. Thank you.

[Xiaoming-Zhao]

Hi, I tested it from scratch, i.e., cloning the repo and running the preprocessing script (not the one you provided) for the freshly-downloaded DAVIS data, but I could not reproduce the issue. I shared my results for the COLMAP part on the dog scene here. You can have a check.

I would recommend first double-checking whether the optical flow and mask results are correct.

Then try several more times with COLMAP. Based on my experience, I do notice that COLMAP has some randomness (and it is hard to make it fully deterministic) for results. Occasionally I can see it fails on the dog scene but I can produce poses for far more than merely 4 images.

As you have noticed, I use COLMAP 3.8. I currently do not have the resources and bandwidth to test it with 3.10. If possible, do you mind trying with version 3.8?

Hope these help.