search

jiawei-ren / dreamgaussian4d

[arXiv 2023] DreamGaussian4D: Generative 4D Gaussian Splatting
https://jiawei-ren.github.io/projects/dreamgaussian4d/
MIT License
501 stars 31 forks source link

Training failure due to the scale difference between the offset and the canonical space gs #17

Open Time-Lord12th opened 1 month ago

Time-Lord12th commented 1 month ago

Hello， Thanks for your great work. Have you ever encountered training failure due to the scale difference between the offset and the canonical space gs? For example, the position range of the canonical space gs is [-0.2, 0.2], while the offset range output by the deformation model is [-1, 1]. So most of the gs will be deformed out of the camera's field of view, causing a large difference between the rendered image and the ground truth, making it impossible for the model to optimize properly. Do you think zero-initialization is helpful to solve this issue? By the way, it seems that initialize_zeros_weights is not used. What's the reason for that? https://github.com/jiawei-ren/dreamgaussian4d/blob/bc229beeda74fcadee167a53b4683b1d7662def3/scene/deformation.py#L199 Thanks!

jiawei-ren commented 1 month ago

Have you ever encountered training failure due to the scale difference between the offset and the canonical space gs?

I have been sticking to the same camera setting so I haven't seen the error before. Will scaling/truncating the XYZ deformation here help?

it seems that initialize_zeros_weights is not used. What's the reason for that?

Zero-init is implemented here should be called here.