Question about `render.py`

[hyoseok1223]

Thank you for great work, I have some questions in render.py

• Below line, say frustum culled, but in this dictionary is not culled any thing, only filter by radii. frustum culling is did in differentiable rasterization right?
• In view space, i think view direction is -Z, so if do frustum culling, we only get -z_near ~ z_far? ( ex - -0.01 ~-100.0 )
  • I see threshold is 0.2 in here, is it in canonical space ( 0~1 ) ? if z=0.2 is not view direction ( i think for example it is thresholded by <-0.2 which mean in frustum
• In render.py, what is viewspace_points mean? really i think it is camera space gaussian xyz point coordinate but value screenspace_points just all zero. why use viewspace_points expression?

https://github.com/graphdeco-inria/gaussian-splatting/blob/f11001b46c5c73a0a7d553353c898efd68412abe/gaussian_renderer/__init__.py#L95C5-L100C28

    # Those Gaussians that were frustum culled or had a radius of 0 were not visible.
    # They will be excluded from value updates used in the splitting criteria.
    return {"render": rendered_image,
            "viewspace_points": screenspace_points,
            "visibility_filter" : radii > 0,
            "radii": radii}

[kwea123]

1. yes culling is done during rasterization
2. z values are in euclidean space
3. screenspace_points is passed to rasterizer function which fills this variable with the image coordinates (uv) of the gaussians

[MobiusLqm]

@kwea123 Hi, Thank your for your explanation. But Why the elements in varible 'screenspace_points' after rendered are still all with zero? Could you explain it more clearly?I'm still confused. Thank you!

[kwea123]

@MobiusLqm oh sorry, that variable is only used to keep the gradient w.r.t. uv, the content itself is still zero. The values are stored in this variable https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L265 but with this code it is very hard to extract them. You need to compute the correct offset from https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L158-L165 then get it from the variable geomBuffer on this line https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/diff_gaussian_rasterization/__init__.py#L86

[MobiusLqm]

@MobiusLqm oh sorry, that variable is only used to keep the gradient w.r.t. uv, the content itself is still zero. The values are stored in this variable https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L265 but with this code it is very hard to extract them. You need to compute the correct offset from https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L158-L165 then get it from the variable geomBuffer on this line https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/diff_gaussian_rasterization/__init__.py#L86

@kwea123 Greate thanks for your explanation, I really appreciate it. I will check it later.

[ShaySheng]

@MobiusLqm oh sorry, that variable is only used to keep the gradient w.r.t. uv, the content itself is still zero. The values are stored in this variable https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L265 but with this code it is very hard to extract them. You need to compute the correct offset from https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L158-L165 then get it from the variable geomBuffer on this line https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/diff_gaussian_rasterization/__init__.py#L86

How exactly does it "retain the gradient with respect to uv"? I was under the impression that means2D is ultimately passed to this section of the code. However, it appears that it isn't utilized there at all. Could you guide me on how to further trace its usage or impact?

[ShaohuaL]

1. yes culling is done during rasterization
2. z values are in euclidean space
3. screenspace_points is passed to rasterizer function which fills this variable with the image coordinates (uv) of the gaussians

Hi！Thank your good tutorials about 3D GS ^-^! I think means2D stores gradients about NDC space, but not image space.

atomicAdd(&dL_dmean2D[global_id].x, dL_dG * dG_ddelx * ddelx_dx); atomicAdd(&dL_dmean2D[global_id].y, dL_dG * dG_ddely * ddely_dy);

[JerryPW]

@MobiusLqm oh sorry, that variable is only used to keep the gradient w.r.t. uv, the content itself is still zero. The values are stored in this variable https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L265 but with this code it is very hard to extract them. You need to compute the correct offset from https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L158-L165 then get it from the variable geomBuffer on this line https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/diff_gaussian_rasterization/__init__.py#L86

How exactly does it "retain the gradient with respect to uv"? I was under the impression that means2D is ultimately passed to this section of the code. However, it appears that it isn't utilized there at all. Could you guide me on how to further trace its usage or impact?

I have the same question with you. Have you figured it out?

[MobiusLqm]

@MobiusLqm oh sorry, that variable is only used to keep the gradient w.r.t. uv, the content itself is still zero. The values are stored in this variable https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L265 but with this code it is very hard to extract them. You need to compute the correct offset from https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L158-L165 then get it from the variable geomBuffer on this line https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/diff_gaussian_rasterization/__init__.py#L86

How exactly does it "retain the gradient with respect to uv"? I was under the impression that means2D is ultimately passed to this section of the code. However, it appears that it isn't utilized there at all. Could you guide me on how to further trace its usage or impact?

I have the same question with you. Have you figured it out?

sorry. I didn't figure it out. this issue is not related with my current project .

[Erhushenshou]

@MobiusLqm oh sorry, that variable is only used to keep the gradient w.r.t. uv, the content itself is still zero. The values are stored in this variable https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L265 but with this code it is very hard to extract them. You need to compute the correct offset from https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L158-L165 then get it from the variable geomBuffer on this line https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/diff_gaussian_rasterization/__init__.py#L86

How exactly does it "retain the gradient with respect to uv"? I was under the impression that means2D is ultimately passed to this section of the code. However, it appears that it isn't utilized there at all. Could you guide me on how to further trace its usage or impact?

I have the same problem. means2D is never used again after passing into the render function.

[FanYaning]

@MobiusLqm oh sorry, that variable is only used to keep the gradient w.r.t. uv, the content itself is still zero. The values are stored in this variable https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L265 but with this code it is very hard to extract them. You need to compute the correct offset from https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L158-L165 then get it from the variable geomBuffer on this line https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/diff_gaussian_rasterization/__init__.py#L86

How exactly does it "retain the gradient with respect to uv"? I was under the impression that means2D is ultimately passed to this section of the code. However, it appears that it isn't utilized there at all. Could you guide me on how to further trace its usage or impact?

I have the same problem. means2D is never used again after passing into the render function.

means2D is not used in the forward function, but _RasterizeGaussians 's backward return grad values as follows:

        grads = (
            grad_means3D,
            grad_means2D,
            grad_sh,
            grad_colors_precomp,
            grad_opacities,
            grad_scales,
            grad_rotations,
            grad_cov3Ds_precomp,
            None,
        ) 

so means2D.grad=grad_means2D.

You can try this example:

import torch

class Exp(torch.autograd.Function):
    @staticmethod
    def forward(ctx, i, j):
        result = i.exp()
        ctx.save_for_backward(result)
        return result

    @staticmethod
    def backward(ctx, grad_output):
        result, = ctx.saved_tensors 
        return (grad_output * result, grad_output * result * 2)

if __name__ == "__main__":
    x = torch.ones(1, requires_grad = True)
    y = torch.ones(1, requires_grad = True)
    y.retain_grad()

    output = Exp.apply(x, y)
    print(output) # tensor([2.7183]

    output.backward((torch.ones(1)))
    print(y.grad)