Wrong value when converting SH to RGB

[jiahaolu97]

Hi, thank you for contributing this repo!

When I use your compute_sh cuda kernel, I found the converted rgb values are not correct ... I am not sure if I misused the cuda kernel:

def compute_sh(
    shs: Float[Tensor, "P C D"],
    view_dirs: Float[Tensor, "P 3"],
    visible: Bool[Tensor, "P 1"] = None,
) -> Float[Tensor, "P 3"]:

# view_dirs : Float[Tensor, "P 3"], Normalized view direction.

Here, I give the view_dirs as the normalized vector of (gaussians.xyz - camera_center), where camera_center is the camera position in world coordinates. To be more specific,

sh_reshape = self.get_feat.reshape(self.get_gaussian_num, 3, -1).contiguous()      # reshape [N, C] -> [N, 3, (sh_degree+1)**2]
viewdirs = self.get_xyz - camera_center.repeat(self.get_gaussian_num, 1)             # [N, 3]
normalized_viewdirs = viewdirs / torch.linalg.norm(viewdirs, dim=-1, keepdim=True)
rgbs = compute_sh_cuda(sh_reshape, normalized_viewdirs, view_visible)

The rgbs computed this way does not has range [0,1] (min value around -1.7 and max value around 2.9).

Here the spherical harmonics coefficients are from the Gaussian model ply trained by the original 3DGS code. The settings are sh_degree = 3 , so spherical harmonics has 3 * (3 + 1) ** 2 = 48 channels.

Anyone has an idea?

[yGaoJiany]

We implement the evaluation of the SH coefficients, rather than getting RGB values from SH. If you need RGB colors, you simply need to do a conversion: sh2rgb = ms.compute_sh(shs.permute(0, 2, 1), direction) rgb = torch.clamp_min(sh2rgb + 0.5, 0.0) Now, display your image and see!

[jiahaolu97]

Thank you @yGaoJiany for clarification!

Indeed we can directly train a RGB or SH model from scratch using your kernels. I found it still has some problems to convert a SH model trained from original 3DGS repo to visualize in this repo, but it is no big problem.

[yGaoJiany]

I tried to render with msplat from 3D Gaussian point cloud, and it seems work. After optimization with orginal 3DGS repo (https://github.com/graphdeco-inria/gaussian-splatting), I did a replacement with msplat (see issue #6), and then run:

python render.py -m "output/gs" --skip_train