High Quality Point Cloud Rendering

[jaidevshriram]

I'm interested in using the PyTorch3D point cloud renderer for visualisation purposes - doesn't need to be differentiable. However, I've found that the rendered images from a point cloud are poorer quality than the image used to generate it. I've tried changing the point radius values around and have reached a PSNR of just ~24 using this renderer compared to the original image. Do you have any suggestions on how to improve the quality of rendered images?

I have also come across these dotted black lines in my rendering when I rotate my camera. I'm not sure why that is. Any suggestions would be appreciated!

[bottler]

These are modelling questions I think, rather than questions about PyTorch3D. For the first, you could look in detail at what the difference is between the two images. When you move the camera, there are pixels which don't hit points corresponding to pixels in the original image, so gaps are expected.

[jaidevshriram]

Thank you! I'm afraid I'm still facing issues on the later problem - I have some black line artifacts on my image (shown below) - seen on the right hand side. I'm using the alpha compositor with a red background colour (hence the red background) which means the black lines aren't empty space really. Increasing the radius does fix the problem but makes the image blurry. Relevant code:

# This is the regular PointsRenderer but also returns the depth map
class PointsRendererCustom(nn.Module):
    def __init__(self, rasterizer, compositor) -> None:
        super().__init__()
        self.rasterizer = rasterizer
        self.compositor = compositor

    def forward(self, point_clouds, **kwargs) -> torch.Tensor:
        fragments = self.rasterizer(point_clouds, **kwargs)

        r = self.rasterizer.raster_settings.radius

        dists2 = fragments.dists.permute(0, 3, 1, 2)
        weights = 1 - dists2 / (r * r)
        images = self.compositor(
            fragments.idx.long().permute(0, 3, 1, 2),
            weights,
            point_clouds.features_packed().permute(1, 0),
            **kwargs,
        )

        # permute so image comes at the end
        images = images.permute(0, 2, 3, 1)

        return images, fragments.zbuf

radius = 0.0045
points_per_pixel = 10
img_size = 512
cameras = # Defined elsewhere

raster_settings = PointsRasterizationSettings(
            image_size=img_size, 
            radius = radius,
            points_per_pixel = points_per_pixel,
            bin_size=0
        )

renderer = PointsRendererCustom(
            rasterizer=PointsRasterizer(cameras=cameras, raster_settings=raster_settings),
            compositor=AlphaCompositor(background_color=(1, 0, 0))
        )

images, depth = renderer(pcd)