Negative value in color

[zhou745]

Recently, I am playing with the Gsp raster, I find out the rendered image can have an absurd negative color value(-1e+23). May I ask why is this happening and how to fix this?

The following is my testing code:

testing the gsp raster

import torch import math from diff_gaussian_rasterization import GaussianRasterizationSettings, GaussianRasterizer import numpy as np from utils.graphics_utils import getWorld2View2, getProjectionMatrix import cv2

def main():

device = torch.device("cuda:0")
#parameters
image_height = 512
image_width = 512
Fx = 1024
Fy = 1024

fovx = 2*math.atan(image_width/(2*Fx))
fovy = 2*math.atan(image_height/(2*Fy))
tanfovx = math.tan(fovx * 0.5)
tanfovy = math.tan(fovy * 0.5)

#raster para
bg_color = torch.tensor([1, 1, 1], dtype=torch.float32, device=device)
scaling_modifier = 1.0

trans = np.array([0.0, 0.0, 0.0])
R = np.array([[1,0,0],
                  [0,1,0],
                  [0,0,1]],dtype=np.float32).transpose(0,1)    #w2c rotation matrix
T = np.array([0.,0.,2.0],dtype=np.float32)
scale = 1.0
zfar = 4.0  # zfar only consider scenery of 4 meters
znear = 0.01

world_view_transform = torch.tensor(getWorld2View2(R, T, trans, scale)).transpose(0, 1).to(device)
projection_matrix = getProjectionMatrix(znear=znear, zfar=zfar, fovX=fovx,
                                             fovY=fovy).transpose(0, 1).to(device)
full_proj_transform = (
    world_view_transform.unsqueeze(0).bmm(projection_matrix.unsqueeze(0))).squeeze(0)
camera_center = world_view_transform.inverse()[3, :3]
#color parameters
active_sh_degree = 0
debug = False
#set the cuda raster
raster_settings = GaussianRasterizationSettings(
    image_height=int(image_height),
    image_width=int(image_width),
    tanfovx=tanfovx,
    tanfovy=tanfovy,
    bg=bg_color,
    scale_modifier=scaling_modifier,
    viewmatrix=world_view_transform,
    projmatrix=full_proj_transform,
    sh_degree=active_sh_degree,
    campos=camera_center,
    prefiltered=False,
    debug=debug
)
rasterizer = GaussianRasterizer(raster_settings=raster_settings)

#create the gsp pc for raster
#a cude of size 1m center at 0,0,0
Temp = 0.2

size = 0.5
step = 128
xs = torch.linspace(-size, size, steps=step)
ys = torch.linspace(-size, size, steps=step)
zs = torch.linspace(-size, size, steps=step)
x, y, z = torch.meshgrid(xs, ys, zs, indexing='xy')
x = x.contiguous().view(-1, 1)
y = y.contiguous().view(-1, 1)
z = z.contiguous().view(-1, 1)
xyz_idx = torch.cat([x, y, z], dim=-1)
xyz_idx = xyz_idx.to(device)

#get xyz
means3D = xyz_idx
#get opacity
opacity = torch.sigmoid(-means3D.norm(dim=-1)).unsqueeze(-1)
scales = (torch.ones_like(means3D)*0.01).to(device)
shs = xyz_idx+0.5
rotations = torch.zeros([scales.shape[0],4],dtype=torch.float32).to(device)
rotations[:,0] = 1.

#some more unsed
means2D = torch.zeros_like(means3D, dtype=means3D.dtype, requires_grad=True, device=device) + 0
cov3D_precomp = None

#color precomp
colors_precomp = None

rendered_image, radii = rasterizer(
    means3D = means3D,
    means2D = means2D,
    shs = shs,
    colors_precomp = colors_precomp,
    opacities = opacity,
    scales = scales,
    rotations = rotations,
    cov3D_precomp = cov3D_precomp)

print(rendered_image.shape)

img = rendered_image.permute(1,2,0).detach().cpu().numpy()*255
print(img.max(),img.min())
filename = "test_img.png"
cv2.imwrite(filename, img)
print("finished rendering")

if name=="main": main()

[yuedajiong]

shs = xyz_idx+0.5 //input valid?

initilized: random , not SfM/colmap output. torch.cat((torch.rand(points.shape[0], 3, 1).float().to(device) / 5.0 + 0.4, torch.zeros((points.shape[0], 3, (self.max_sh_degree + 1) ** 2 -1)).float().to(device)), dim=-1)

[zhou745]

shs = xyz_idx+0.5 //input valid?

initilized: random , not SfM/colmap output. torch.cat((torch.rand(points.shape[0], 3, 1).float().to(device) / 5.0 + 0.4, torch.zeros((points.shape[0], 3, (self.max_sh_degree + 1) ** 2 -1)).float().to(device)), dim=-1)

Yeal its valid, the range of shs value is from 0 to 1.

I am actually looking at a cube that is 1m long from 2 meters away (I also tried T = np.array([0.,0.,6.0],dtype=np.float32) 6 m away, it's also producing negative color). The cube consists of 128**3 gaussian points.

Also, I find out that with fewer points, say 64**3 points' the render is normal.

[yuedajiong]

我感觉我事实上已经指出你代码的问题了，你还没有意识到。 shs = xyz_idx+0.5 你这里是将点云的立体的高斯点位置给移位了一下就作为SH系数。 逻辑上这里需要顶一个（Point, 16)的矩阵（这是缺省，如果你要搞的比较小，要同步修改sh_degree从0~3或者更大）

double check your code?

print(rendered_image.detach().cpu().max(),rendered_image.detach().cpu().min(), ' ', pc.get_features.detach().cpu().max(), pc.get_features.detach().cpu().min())

tensor(0.5389) tensor(0.0968) tensor(1.2998) tensor(-1.7725) tensor(0.5321) tensor(0.0905) tensor(1.2973) tensor(-1.7750) tensor(0.5432) tensor(0.0893) tensor(1.2948) tensor(-1.7766) tensor(0.5544) tensor(0.0551) tensor(1.2950) tensor(-1.7779) tensor(0.5095) tensor(0.0669) tensor(1.2964) tensor(-1.7790) ...

tensor(0.6711) tensor(0.) tensor(1.7638) tensor(-1.8185) tensor(0.6348) tensor(0.) tensor(1.7652) tensor(-1.8185) tensor(0.8121) tensor(0.) tensor(1.7665) tensor(-1.8185) tensor(0.8017) tensor(0.) tensor(1.7677) tensor(-1.8185)

[zhou745]

我感觉我事实上已经指出你代码的问题了，你还没有意识到。 shs = xyz_idx+0.5 你这里是将点云的立体的高斯点位置给移位了一下就作为SH系数。 逻辑上这里需要顶一个（Point, 16)的矩阵（这是缺省，如果你要搞的比较小，要同步修改sh_degree从0~3或者更大）

double check your code?

print(rendered_image.detach().cpu().max(),rendered_image.detach().cpu().min(), ' ', pc.get_features.detach().cpu().max(), pc.get_features.detach().cpu().min())

tensor(0.5389) tensor(0.0968) tensor(1.2998) tensor(-1.7725) tensor(0.5321) tensor(0.0905) tensor(1.2973) tensor(-1.7750) tensor(0.5432) tensor(0.0893) tensor(1.2948) tensor(-1.7766) tensor(0.5544) tensor(0.0551) tensor(1.2950) tensor(-1.7779) tensor(0.5095) tensor(0.0669) tensor(1.2964) tensor(-1.7790) ...

tensor(0.6711) tensor(0.) tensor(1.7638) tensor(-1.8185) tensor(0.6348) tensor(0.) tensor(1.7652) tensor(-1.8185) tensor(0.8121) tensor(0.) tensor(1.7665) tensor(-1.8185) tensor(0.8017) tensor(0.) tensor(1.7677) tensor(-1.8185)

我测试的代码中用的是active_sh_degree = 0, 所以这里应该是一个 Nx3的矩阵吧？ 还是说依然需要一个Nx16的矩阵？ 不过就像我说的， 当高斯场的点密度比较小， 从1283/1m3 降到643/1m3 这个bug就消失了， 我怀疑应该是渲染的时候与点的数量相关的地方，超出某个数组的界了。

补充一下： 我的这个测试代码是相当于我手动设计了一个正方体，这个正方体的每个高斯球的颜色是自己的坐标+0.5（保证颜色在0-1，然后opacity 是高速球到原点的距离的负sigmoid）

[yuedajiong]

前面读的代码，如果我没有记错：active_sh_degree是一个训练中变化的量，是一个从0阶段递增到你最大的sh_degree。 对应到就是cuda中那个函数，那个是一层一层往里面调用的。 开始只有1/16起作用。 如果你的场景足够简单，用不了那么长的系数，你把sh_max设置为0,就只用第一位，就是表达能力弱一点。 如果你按照缺省参数，训练好了，这个active就应该到最大。 如果修改过render，其实只是render，那么纯渲染其实不需要这个参数，拉满16用。

按照deg: 需要的长度（index）： 0：0 1： 123 2： 45678 3： 9~15 更大，没用。

“我怀疑应该是渲染的时候与点的数量相关的地方，超出某个数组的界了。” 有可能，我前面也提过一个类似的issue。 https://github.com/graphdeco-inria/gaussian-splatting/issues/472 我不肯定问题一样。

[pknmax]

@zhou745 @yuedajiong how to get final color of particular gaussian given there are gaussians at the back of that gaussian?