search

threestudio-project / threestudio

A unified framework for 3D content generation.
Apache License 2.0
6.29k stars 478 forks source link

Issue with Gradient Anomalies in 3D Gaussian Optimization Using ThreeStudio #345

Open chensjtu opened 11 months ago

chensjtu commented 11 months ago

I've encountered an issue while trying to optimize 3D Gaussian using ThreeStudio. The gradient behavior seems abnormal. In the 3D Gaussian optimization process, the gradient of visible_points is crucial for determining whether to clone and split Gaussian points. However, during my tests in PyTorch Lightning, the gradients of visible_points are unusually large and often result in NaN values. Is this behavior expected? How can I modify my code to achieve cloning and splitting similar to the original version of the code? the grad:


        [    nan,     nan,  0.0000],
        [-0.1534,  0.1053,  0.0000],
        ...,
        [ 0.0000,  0.0000,  0.0000],
        [ 0.7816,  0.5890,  0.0000],
        [    nan,     nan,  0.0000]], device='cuda:0')```
DSaurus commented 11 months ago

Hi @chensjtu ,

I haven't checked yet, but I'd like to know which branch you are using. I have another branch called saurus/gaussian for Gaussian Splatting. Will this branch encounter the same issues?

chensjtu commented 11 months ago

Actually, I use the codebase of GaussianDreamer, which is developed on threestudio. The visible points' grad is abnormal, causing the number of points doubled every clone operation.

DSaurus commented 11 months ago

That's interesting. I have also noticed the number of points will be doubled every time. I will check it.

DSaurus commented 11 months ago

@chensjtu I haven't encountered the same issue in my implementation. You could consider using threestudio-3dgs extension.

debug

chensjtu commented 11 months ago

wow! That's amazing! Many thanks for the effort made on this issue! I'll check the differences of the code. Maybe I can find what's wrong.

tuning12 commented 1 month ago

hi, have you found the reason?