Problems regarding pose

[JubSteven]

Hello, thanks for your great work! I want to ask about the transformation matrix provided in meta. Is the transformation matrix the relative transform from the raw-scans (objects) in canonical coordinates to the objects in camera coordinates?

[JubSteven]

After some empirical experiments, I found that to transform the mesh from raw_scan to that corresponding to blender_render, you need to scale the mesh first using meta["scale"], and then transform the mesh by 90 degrees around the x axis. After that, apply pose[:, 1:3] *= -1 according to the dataloader and then take its inverse.

[Bekci]

Do you have the code to align the raw_scan with blender_render?

[JubSteven]

Do you have the code to align the raw_scan with blender_render?

Currently I use the following code

def get_obj_transf(self, idx):
    meta = self._load_meta(idx)
    cam_extr = np.array(meta['transform_matrix'])
    cam_extr[:, 1:3] *= -1  # Alter the coordinates

    cam_extr = np.linalg.inv(cam_extr)
    return cam_extr

def get_obj_verts_can(self, idx):
    obj_mesh = self._load_mesh(idx)
    return np.asarray(obj_mesh.vertices).astype(np.float32)

def get_obj_verts_3d(self, idx):
    verts_c = self.get_obj_verts_can(idx)
    obj_transf = self.get_obj_transf(idx)
    verts_3d = SE3_transform(verts_c, obj_transf)
    return verts_3d

def _load_mesh(self, idx):
    sample_info = self.sample_idx[idx]
    meta = self._load_meta(idx)
    obj_name = sample_info.split('/')[0]
    obj_cat = "_".join(obj_name.split('_')[:-1])

    self.base_mesh_path = os.path.join(self.data_root, "OmniObject3D", "data", "raw_scans")
    mesh_path = os.path.join(self.base_mesh_path, obj_cat, obj_name, "Scan", "Scan.obj")
    obj_mesh = trimesh.load_mesh(mesh_path)
    obj_mesh.apply_scale(meta["scale"])

    _coord_trans = np.array(
        [
            [1, 0, 0, 0],
            [0, 0, -1, 0],
            [0, 1, 0, 0],
            [0, 0, 0, 1],
        ],
        dtype=np.float32,
    )

    obj_mesh.apply_transform(_coord_trans)

    return obj_mesh

Project the obj_verts_3d using cam_intr back into the image should give the correct result.

[Bekci]

Thank you! The code helped me a lot.