Running on existing datasets like SRN

[subClassy]

Hi, Really interesting paper. I was trying to run the notebook on SRN data which contains pose information and camera intrinsic. But using that pose seems to be giving incorrect results in visualization. In this, I am taking the inverse of the rotation matrix available with these images. Any help would be really appreciated!

Thanks

[jasonyzhang]

Hi,

SRN uses a different camera convention than Pytorch3D. That is, even if you took the SRN car mesh and the provided SRN camera pose and rendered them using Pytorch3D, the viewpoint would be completely off.

I would start by doing the following transformations on the provided camera pose:

    def load_pose(pose_path):
        """Loads the camera pose given the path to the extrinsics file
​
        Args:
            pose_path(str): The path corresponding to the camera pose
​
        Returns:
            np.array: The 4x4 extrinsics matrix
        """
        R_rel_tr = torch.FloatTensor([
            [0, -1,  0],
            [0,  0, -1],
            [1,  0,  0]
        ]).T
        axes_flip = torch.diag(torch.tensor([-1, -1, 1.0]))
        pose = torch.from_numpy(
            np.loadtxt(pose_path, dtype=np.float32).reshape(4, 4)
        )
        pose = torch.linalg.inv(pose)  # "camera2world" (srn dataset) -> "world2camera" (pytorch3d)
        pose[:3, :3] = axes_flip @ (pose[:3, :3] @ R_rel_tr) @ axes_flip
        return pose

If these modifications are enough, please provide a notebook with the SRN data you are trying to load (preferably with camera transform + original mesh).

Best, Jason