pointrix-project / msplat

A modular differential gaussian rasterization library.
Other
170 stars 10 forks source link

problem when rendering gaussian from its original repo #10

Closed CA-TT-AC closed 5 months ago

CA-TT-AC commented 5 months ago

I am trying to use msplat to render 3DGaussian in its original repo (GS). When transferring its original parameters to msplat, I met several questions and don't know why them can not match. I propose several potential problem:

  1. There is difference between world_view_transform in GS and extr in msplat.
  2. The preprocess of RGB feature can not be used in msplat.

Here is my code:

    # generate RGB feature
    shs_view = gaussians.get_features.transpose(1, 2).view(-1, 3, (gaussians.max_sh_degree+1)**2)
    dir_pp = (gaussians.get_xyz - viewpoint_cam.camera_center.repeat(gaussians.get_features.shape[0], 1))
    dir_pp_normalized = dir_pp/dir_pp.norm(dim=1, keepdim=True)
    sh2rgb = eval_sh(gaussians.active_sh_degree, shs_view, dir_pp_normalized)
    colors_precomp = torch.clamp_min(sh2rgb + 0.5, 0.0)
    print(colors_precomp.shape)

    # msplat render
    xyzs = gaussians.get_xyz
    scales = gaussians.get_scaling
    rotations = gaussians.get_rotation
    opacity = gaussians.get_opacity
    feats = gaussians.get_features
    intr = torch.tensor([viewpoint_cam.FoVx, viewpoint_cam.FoVy, viewpoint_cam.image_width / 2, viewpoint_cam.image_height/ 2], dtype=torch.float32, device="cuda")
    extr = viewpoint_cam.world_view_transform
    W = viewpoint_cam.image_width
    H = viewpoint_cam.image_height
    image_msplat = msplat.rasterization(
xyzs, scales, rotations, opacity, colors_precomp, intr, extr, W, H, 0)       
    # test visulization
    if iteration % 1000 == 0:
        # this is right
        cv2.imwrite("comp/msplat_{}.jpg".format(iteration), image_msplat.permute(1,2,0).cpu().detach().numpy()*255)
        # this is wrong
        cv2.imwrite("comp/gs_{}.jpg".format(iteration), image.permute(1,2,0).cpu().detach().numpy()*255)

The rendering output of msplat is like this: image

Thanks very much if you can give any cue to this problem!!

yGaoJiany commented 5 months ago

For Q1, please see https://github.com/pointrix-project/msplat/issues/6#issuecomment-2116971018 or https://github.com/yGaoJiany/gaussian-splatting-msplat. For Q2, the description is a bit confusing. Could you provide more details?

CA-TT-AC commented 5 months ago

This problem has been solved by using code from https://github.com/yGaoJiany/gaussian-splatting-msplat!! Thanks for your help!