Have you ever tested your method on the Shiny Blender Dataset (provided by Ref-NeRF)?

[YessionCC]

Shiny Blender Dataset contains some reflective objects and is frequently used for evaluating inverse rendering. If you have conducted relative tests, could you kindly provide the corresponding configurations or some results? Your assistance would be greatly appreciated.

[lzhnb]

We did not consider the reconstruction of reflective (glossy) objects when doing this project, so if you use our method on this dataset, you will find some problem (e.g. glossy surface is depressed). For this case, GaussianShader specializes in processing related datasets, and you can refer to it. In addition, we found that GaussianShader does not perform well on the TensoIR, which almost consists of diffuse material, so GS-IR and GaussianShder each draw on their own strengths.

But we have done some exploration in this part and there are some relevant experiences with you:

1. the geometric reconstruction of reflective/glossy objects must make normal learning related to the color reconstruction strongly;
2. the deferred shading we use is not good as handling this aspect. Specifically, the normal results obtained by deferred shading generally do not satisfy $\Vert \mathbf{n} \Vert = 1$, especially for transparent materials, which is not a problem in forward shading.

I suggest that this problem can be solved if you change GS-IR's deferred shading to forward shading (i.e. PBR shades each partical and inputs the shaded color as colors_precomp).

[YessionCC]

Thanks for your kind reply. You really do a excellent job. You use the physically based rendering model, which should support reflection reconstruction theoretically. Can I make a conclusion that normals are an important factor influencing the inverse problem, as I observe that for smooth objects, the distribution of 3D Gaussians are very disorderly. I have tested GaussianShader, but their results are not quite satisfactory. I guess that compared to previous SDF-based methods, 3DGS is relatively under-constrained. I am utilizing 3DGS for modeling racecars, and I look forward to the day when this issue can be effectively resolved.

[lzhnb]

Thanks for your kind reply. You really do a excellent job. You use the physically based rendering model, which should support reflection reconstruction theoretically. Can I make a conclusion that normals are an important factor influencing the inverse problem, as I observe that for smooth objects, the distribution of 3D Gaussians are very disorderly. I have tested GaussianShader, but their results are not quite satisfactory. I guess that compared to previous SDF-based methods, 3DGS is relatively under-constrained. I am utilizing 3DGS for modeling racecars, and I look forward to the day when this issue can be effectively resolved.

Yes, your conclusion is right. 3DGS is closer to a contiguous energy field and is not good at tackling hard and solid geometry. The recent 2DGS is promising to fundamentally solve this problem

[YessionCC]

Thanks. btw, I have a suggestion for your code implementation.

In the function recon_occlusion, you perform importance sampling to compute the ambient occlusion. Considering that this process is only for the diffuse component, you can precompute it in the bake stage (save the SH of occlusion irradiance instead of the occlusion itself), and directly query it using the normal direction in runtime. By doing so, the normal can also acquires meaningful gradients when training.