hbb1 / 2d-gaussian-splatting

[SIGGRAPH'24] 2D Gaussian Splatting for Geometrically Accurate Radiance Fields
https://surfsplatting.github.io
Other
2.11k stars 144 forks source link

Poor reconstruction quality for some very cluttered areas #133

Open CanCanZeng opened 3 months ago

CanCanZeng commented 3 months ago

Hello, I found that 2DGS may reconstruct some very cluttered areas poorly. If you observe the distribution pattern of the point cloud, the point cloud in these areas is very sparse. The original 3DGS also has this problem, so someone proposed the AbsGS algorithm to improve the densification strategy. I encountered difficulties when trying to transplant this improvement into 2DGS, because the rendering of 2DGS is different from 3DGS, and the gradient of the 2D projection point is 0 in most cases. Does the author have any good suggestions for this problem?

Note the lower right corner of the picture below image

hbb1 commented 3 months ago

Hi, indeed 2DGS's densitification metric relies on the gradient of the 3D point but I believe that the absGrad can be implant into the 3D gradient as well.

CanCanZeng commented 3 months ago

Sorry, I don't understand what you mean. Do you mean that the gradient of viewspace_point_tensor in 2DGS is actually the gradient of loss relative to the 3D point?

hbb1 commented 3 months ago
image

You may check the code for further adaption. https://github.com/hbb1/diff-surfel-rasterization/blob/main/cuda_rasterizer/backward.cu#L639-L640

CanCanZeng commented 3 months ago

I'm sorry that I missed this detail when I read the paper. Thank you for your patient reply.

hwanhuh commented 3 months ago

Like AbsGS, you can add homodirectional gradient accumulation to the backward kernel. Then, screenspace_points (defined in gaussian_renderer) must be modified as follows.

-edited

screenspace_points = torch.zeros((pc.get_xyz.shape[0], 4), dtype=pc.get_xyz.dtype, requires_grad=True, device="cuda") + 0

Results: (top: org 2D GS, bottom: w/ homodirection gradient) image

hbb1 commented 3 months ago

Exactly. I add some hightlights.

image
CanCanZeng commented 3 months ago

Hi @hwanhuh thank you for your advice. Your result looks very good, but I tried the modification you made modified and the result is worse than the original version. May I ask why this is? In fact, I have made almost the same modifications as you and found that the result is incorrect. The value of grads_abs is smaller than the value of grads, so I came to consult the author.

CanCanZeng commented 2 months ago

@hwanhuh I also tried to transplant the MCMC algorithm into the 2DGS algorithm, which can also solve the problem of blurry cluttered areas. However, the MCMC algorithm introduces a new problem, which is that in areas with the sky as the background, if the foreground is a structure full of details such as tree branches, the reconstruction quality of this part will become blurry. Do you have a solution to this problem?

https://github.com/ubc-vision/3dgs-mcmc/issues/17

hwanhuh commented 2 months ago

@CanCanZeng I plan to publish the 2D GS application project of AbsGS / MCMC, including other implementation details, on my GitHub, but not immediately. I will let you know when the code is organized.

Early on in my experiments applying MCMC, I observed a similar trend. ( In my opinion) This is because, since the relocation alone does not provide an appropriate update to the low-density region, the noise term needs to be carefully designed for MCMC. Directly applying MCMC's noise design to 2D GS is improper for me.

CanCanZeng commented 2 months ago

It's really kind of you to share your findings @hwanhuh