search

ingra14m / Deformable-3D-Gaussians

[CVPR 2024] Official implementation of "Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction"
https://ingra14m.github.io/Deformable-Gaussians/
MIT License
947 stars 55 forks source link

Question about the method: Why the detach (stop gradient) operation during training? #80

Open preacherwhite opened 1 week ago

preacherwhite commented 1 week ago

First thanks for the great work and the clean repository. It has been pretty easy to use. I have one question regarding the line 97 in https://github.com/ingra14m/Deformable-3D-Gaussians/blob/main/train.py d_xyz, d_rotation, d_scaling = deform.step(gaussians.get_xyz.detach(), time_input + ast_noise) In the paper there is also a stop gradient operation indicated in this step. I found there isn't an explanation on this choice in both the main paper and supplementary. Why do we stop gradients from the deformation network to the gaussians?

ingra14m commented 1 week ago

Hi, thanks for the interest.

We want the position gradient of 3DGS to be clean. That is, the update of the canonical Gaussian position should only come from the RGB loss, rather than also involving the deformation field branch. Theoretically, this can separate the learning of the deformation field from the canonical Gaussian, making joint training possible.