search

facebookresearch / co3d

Tooling for the Common Objects In 3D dataset.
Other
963 stars 75 forks source link

Unit of depth maps w.r.t. camera extrinsics #50

Closed yashbhalgat closed 2 years ago

yashbhalgat commented 2 years ago

Hi @davnov134, thanks for releasing the v2 of the dataset! It's very useful.

I am trying to perform depth-based warping between two images of the same sequence. What are the "units" of the depth maps with respect to the camera extrinsics (translations)? I am, so far, getting incorrect warpings which indicates that maybe the depths and camera poses have a different scale? Can you please advise?

I am using the _load_16big_png_depth function to load depths. And have already taken into account the scale_adjustment attribute.

Many thanks!

davnov134 commented 2 years ago

Hello, if you are interested in implementing depth-based warping I strongly recommend using the following code: https://github.com/facebookresearch/co3d/blob/f7bf598d4dcff7c5145b1347ea48c802a2e7c134/co3d/utils/dbir_utils.py#L110

It takes a FrameData object, converts it to a pointcloud, and later renders to a new view (basically a classic forward depth-based warping).

The key function to look for is get_rgbd_pointcloud which implements the correct way of unprojecting points from a depth map to a point cloud. https://github.com/facebookresearch/pytorch3d/blob/a393cccf6ae95f7510371c1dafe42d403a2b1d24/pytorch3d/implicitron/tools/point_cloud_utils.py#L23

davnov134 commented 2 years ago

But to clarify, the scale is consistent for both point clouds and the corresponding cameras.

yashbhalgat commented 2 years ago

Thanks for the clarifications! After spending some time with the code yesterday, I indeed found the dbir_utils quite useful.