Inconsistent results after point rasterization and unprojection

[yzhang-gh]

🐛 Bugs / Unexpected behaviors

Hi Pytorch3D team,

I was trying to project a point cloud to another camera view and then unproject it back to get a new point cloud which thus has no occlusion (as in https://github.com/facebookresearch/pytorch3d/issues/906#issuecomment-969439859). However, the two point clouds do not seem to be close enough to each other. Would be happy to see your advice. Thanks!

Might be related: #697 (point cloud mismatch), #906 (unprojection), #1147 (rasterization).

Instructions To Reproduce the Issue:

Here is a simple piece of code to reproduce the problem (attached a data.zip)

import torch
from pytorch3d.io import IO
from pytorch3d.renderer.cameras import PerspectiveCameras
from pytorch3d.renderer.points.rasterizer import PointsRasterizationSettings, PointsRasterizer
from pytorch3d.structures import Pointclouds

H, W = 1080, 1920
R, T, intri, pcd_tensor = torch.load("data.pt")

device = "cuda"
io = IO()

pcd = Pointclouds(pcd_tensor)
io.save_pointcloud(pcd, "origin.ply")

focal_length = torch.Tensor((intri[0, 0].item(), intri[1, 1].item()))[None].to(device)
principal_point = torch.Tensor((intri[0, 2].item(), intri[1, 2].item()))[None].to(device)
cameras = PerspectiveCameras(
    focal_length=focal_length, principal_point=principal_point,
    in_ndc=False, R=R, T=T, image_size=((H, W),), device=device,
)
rasterizer = PointsRasterizer(cameras, PointsRasterizationSettings((H, W)))

## Rasterize
frags = rasterizer(pcd)
depth = frags.zbuf[..., 0]  ## (B, H, W)

## Unproject
# messy coordinate system convertion... but somehow looks alright in the end
X, Y = torch.meshgrid(torch.arange(W), torch.arange(H), indexing="ij")
depth_map = depth[0].clone().transpose(1, 0).flip([0, 1])
camera_xyz = torch.concat((X[..., None].to(device), Y[..., None].to(device), depth_map[..., None]), axis=-1)

world_xyz = cameras.unproject_points(camera_xyz)
world_xyz = world_xyz.reshape([pcd_tensor.shape[0], -1, 3])
io.save_pointcloud(Pointclouds(world_xyz), "unproj.ply")

see the results

yellow: original; red: unprojected

[bottler]

There are some details which are easy to miss in doing the unprojection. Can you use the function get_rgbd_point_cloud from pytorch3d.implictron.tools.point_cloud_utils (see here) instead of your own version?

[yzhang-gh]

Thanks for the speedy reply! I'll try it and get back to you tomorrow.

[yzhang-gh]

Hi @bottler, the get_rgbd_point_cloud function works perfectly well, thank you.