What's the role of depth information in rgbd pipeline?

[Richard-coder]

I am learning rgbd pipeline from 'kapture-localization/pipeline/examples/run_7scenes_rgbd.sh'， create_3D_model_from_depth.py, kapture_pipeline_mapping, kapture_colmap_build_map.py。

I notice that create_3D_model_from_depth.py can read depth image and save 2D point's depth information. However, in kapture_colmap_build_map.py will do triangulation by following code. I thik 'triangulation' not in skip_list is True in rgbd pipeline.

I'm confused that if I can get depth information from depth image, why I need to do triangulation? What's the role of depth information in rgbd pipeline?

Thank you!

    if kapture_data.trajectories is not None:
        # Generate priors for reconstruction
        os.makedirs(priors_txt_path, exist_ok=True)
        if 'priors_for_reconstruction' not in skip_list:
            logger.info('Step 3: Exporting priors for reconstruction.')
            colmap_db = COLMAPDatabase.connect(colmap_db_path)
            database_extra.generate_priors_for_reconstruction(kapture_data, colmap_db, priors_txt_path)
            colmap_db.close()

        # Point triangulator
        reconstruction_path = path.join(colmap_path, "reconstruction")
        os.makedirs(reconstruction_path, exist_ok=True)
        if 'triangulation' not in skip_list:
            logger.info("Step 4: Triangulation")
            colmap_lib.run_point_triangulator(
                colmap_binary,
                colmap_db_path,
                get_image_fullpath(kapture_path),
                priors_txt_path,
                reconstruction_path,
                point_triangulator_options
            )

[yocabon]

Hi, create_3D_model_from_depth replaces kapture_colmap_build_map when depth information is available (more accurate reconstruction). In run_7scenes_rgbd.sh, both the normal pipeline (rgb) and the depth pipeline (rgbd) are run, and evaluated. Notice that at the end, there are two results: pycolmap-localize_rgbd and pycolmap-localize_triangulation.

  # evaluate results
  kapture_evaluate.py -v debug -f \
    -i \
    ${EXP_PATH}/pycolmap-localize_rgbd \
    ${EXP_PATH}/pycolmap-localize_triangulation \
    --labels rgbd triangulation \
    -gt ${DATASETS_PATH}/${SCENE}/query \
    -o ${EXP_PATH}/eval

[Richard-coder]

Thank you for your reply!

I have an another quetion. I notice that in create_3D_model_from_depth.py use VoxelGrid to filter 3d point. If there are tow or more 2d points map to one voxel(3d point), it only use ther first 2d point's 3d information. Have you ever try merge 3d information from defferent frame by averaging or other way?

Thank you!

[yocabon]

No, I don't think we tried to refine the 3d point position from the list of observations.