Aligning problem when doing evaluation on the replica dataset

[Isabel-jin]

Hi，thanks for your great work! I'm trying to doing evaluation on replica using your scripts in replica_eval/eval_recon.py. I did the virualization and found I had problem in aligning the reconstruction mesh with the groundtruth mesh in rotation. I tried using the ICP algorithm offered in your code and tested several different threshlods. But due to the fact that the shape of the replica mesh is like a symethical cuboid, the two meshes always differ in rotation by 180 degrees on one axis. Did you encounter similar problems? Is there any solution in additon to adjust the parameters?

def get_align_transformation(mesh_rec, mesh_gt):
    """
    Get the transformation matrix to align the reconstructed mesh to the ground truth mesh.
    """    
    o3d_gt_mesh = o3d.geometry.TriangleMesh(
        vertices=o3d.utility.Vector3dVector(mesh_gt.vertices),
        triangles=o3d.utility.Vector3iVector(mesh_gt.faces)
    )
    o3d_rec_mesh = o3d.geometry.TriangleMesh(
        vertices=o3d.utility.Vector3dVector(mesh_rec.vertices),
        triangles=o3d.utility.Vector3iVector(mesh_rec.faces)
    )
    o3d_gt_pc = o3d.geometry.PointCloud(points=o3d_gt_mesh.vertices)
    o3d_rec_pc = o3d.geometry.PointCloud(points=o3d_rec_mesh.vertices)
    trans_init = np.eye(4)
    threshold = align_threshold
    reg_p2p = o3d.pipelines.registration.registration_icp(
        o3d_rec_pc, o3d_gt_pc, threshold, trans_init,
        o3d.pipelines.registration.TransformationEstimationPointToPoint(),
        o3d.pipelines.registration.ICPConvergenceCriteria(max_iteration = 3000)
)
    transformation = reg_p2p.transformation
    return transformation

def calc_3d_metric(rec_meshfile, gt_meshfile, align=True):
     ...
     if align:
            transformation = get_align_transformation(mesh_rec, mesh_gt)
            mesh_rec = mesh_rec.apply_transform(transformation)

[RemiChierchia]

Hi @Isabel-jin I've encountered the same issue. I just quickly tried to comment out lines 122-124 and that did the trick for me. trimesh.bounds.oriented_bounds seems to change the coordinate system with respect to the previously extracted mesh and gt mesh. But let's wait for the official response.

def calc_3d_metric(rec_meshfile, gt_meshfile, align=False):
    """
    3D reconstruction metric.

    """
    mesh_rec = trimesh.load(rec_meshfile, process=False)
    mesh_gt = trimesh.load(gt_meshfile, process=False)

    if align:
        transformation = get_align_transformation(rec_meshfile, gt_meshfile)
        mesh_rec = mesh_rec.apply_transform(transformation)

    # found the aligned bbox for the mesh
    # to_align, _ = trimesh.bounds.oriented_bounds(mesh_gt)
    # mesh_gt.vertices = (to_align[:3, :3] @ mesh_gt.vertices.T + to_align[:3, 3:]).T
    # mesh_rec.vertices = (to_align[:3, :3] @ mesh_rec.vertices.T + to_align[:3, 3:]).T

    min_points = mesh_gt.vertices.min(axis=0) * 1.05
    max_points = mesh_gt.vertices.max(axis=0) * 1.05