Rendered depths seem weird in demo.ipynb

[Master-cai]

Thanks for your wonderful work!

When I running the demo.ipynb, the output rendered depths seem weird and different from the original output of the notebook. Any idea? Btw, I create a new envonment following your guidelines in cuda 12.1.

Thank!

[RaymondJiangkw]

Thanks for your interest! This is because I change some code after writing the notebook and forget to update it… sorry for that.

The fix should be calling ‘set_up_scale_and_shift’ on every training camera as in the ‘train.py’ for initialization. I will update the notebook soon as well.

[RaymondJiangkw]

Should be fixed now.

[Master-cai]

@RaymondJiangkw sorry to bother you again, I clone the latest code(09fb5a) again and reinstall the env, the depths seem normal but the rendered images are all black.

[RaymondJiangkw]

This is weird. Could you check the specific value of the rendered image? Also the value of rendered screen space coordinates. Since the depth shows up, the image should not be all black.

[Master-cai]

rendered image is all zeros

rendered screen space seems normal

@RaymondJiangkw

[RaymondJiangkw]

Since this problem does not show up in my system, I may need you to change some code for debugging... sorry for the inconvenience.

Could you change the line L476 in submodules/diff-gaussian-rasterization-appr-surface/cuda_rasterizer/forward.cu from out_color[ch * H * W + pix_id] = C[ch] + T * bg_color[ch]; into out_color[ch * H * W + pix_id] = T;, and then reinstall the rasterizer and check the output of rendered image?

This is to check the remaining transmittance for the image rendering. If the results are close to 0, it then seems to be the color compute problem. If the results are close to 1, it is then very weird...

[Master-cai]

@RaymondJiangkw Thanks for your quick reply! I change the code and reinstalled the rasterizer, the image is still black.

[RaymondJiangkw]

It means that the image rendering correctly aggregates the intersected gaussian kernels... but still I am also not sure why the radiance is zero. May I know your environment? like, which card are you using?

[Master-cai]

OS: Ubuntu 22.04.3 LTS, A100 gpu, cuda 12.1, Driver Version: 470.161.03

I only change the pytorch-cuda=11.8 to pytorch-cuda=12.1 in the environment.yml, others keeps un-changed.

It's really weird as in the first comment, the rendered image seems normal. @RaymondJiangkw

[RaymondJiangkw]

Yes, indeed. I think I have similar environment, and I will test that.

[Master-cai]

@RaymondJiangkw Thank your so much! I export my conda env, hoping helpful for you to find the problem. Please ignore the .txt postfix. cogs.yml.txt

[RaymondJiangkw]

Hi, sorry for the problem. Seems that the problem arises after I specify the newest glm in the rasterizer. If you check, you will see that the official gaussian splatting rasterizer also gives complete black images, and if you use convert_SHs_to_color = True, the images will show out.

The solution is to replace one line in the forward.cu for both rasterizers, which I have committed the changes. Please pull the changes and reinstall both rasterizers to see whether the problem still exists. Hopefully, there aren't additional bugs after specifying the newest glm...

If you encounter other problems, feel free to let me know!

[Master-cai]

Thanks! The rendering goes well now.

I have another question. I'm wondering how can you initialize the first view so good, it seems you do not train the first view. And I think it's the cornerstone for the following optimization, since you compute matches based on the rendered image. In original gaussian splatting, even the colmap pose are given, it can not directly render such high quality image without training. I have read the supp and known that you initialize the scaling and the center of the gaussian ellipsoid to a computed value instead of the average distance between neighbors (proposed in vanilla gaussians).

I want to ask what's the key success factor ? Can you introduce how to achieve such high quality initial rendering in more detail? Can it be applied to vanilla gaussians?

Thanks again! @RaymondJiangkw

[RaymondJiangkw]

Hi, this is simply the advantage given by the explicit primitives. It is also what ‘construct’ means in the title. :)

Since the gaussian kernels are explicit, we can simply construct a gaussian for each pixel, which guarantees that the rendered image and depth match with the provided rgb image and monocular depth.