How to visualize the pointcloud of the predicted image?

[Winter-Dry]

Thanks for your wonderful work!

When I inference with UniDepth, I want to see the visualization of the output pointcloud, could you please release the related scripts?

Thanks again!

[lpiccinelli-eth]

Thank you for your appreciation!

The output predictions["points"] contains the x,y,z locations in OpenCV coordinates system, i.e. east,south,fwd.

To visualize them you can export the .ply file, by flattening the H,W dimension of xyz array and the RGB image and using save_file_ply from unidepth/utils/visualization.py, then use open3d to visualize it. A possible example could be (di not test it):

# load model and inputs as in demo.py
rgb = np.array(Image.open("assets/demo/rgb.png"))
rgb_torch = torch.from_numpy(rgb).permute(2, 0, 1)
intrinsics_torch = torch.from_numpy(np.load("assets/demo/intrinsics.npy"))
predictions = model.infer(rgb_torch, intrinsics_torch)

# save pointcloud
from unidepth.utils.visualization import save_file_ply
filename = "myply.ply"
H, W = rgb.shape[:2]
points = predictions["points"].squeeze().permute(1,2,0).numpy().reshape(H*W, 3)
rgb = rgb.reshape(H*W, 3)
save_file_ply(rgb, points, filename)

# visualize
import open3d as o3d
pcd = o3d.io.read_point_cloud(filename)
flip_y_transform = np.array([[1.,0,0,0],[0,-1,0,0],[0,0,-1,0],[0,0,0,1]])
pcd = pcd.transform(flip_y_transform)
o3d.visualization.draw_geometries([pcd])

We did not want to add dependencies with open3d or pytorch3d as it may not be fully supported on some architecture, that is why there is no script to visualize them directly. However, we can have a separate demo for it and let the user install the dependency in case that demo is utilized.

[Winter-Dry]

Thanks for your kindness!