search

fatPeter / mini-splatting

104 stars 10 forks source link

Backpropagate through the render_depth function #7

Closed yzhang-gh closed 1 month ago

yzhang-gh commented 1 month ago

Hi, this work looks amazing.

May I ask whether it is possible to retain the gradients of the rendered depth from the render_depth function? If it is possible, could you give me a little guidance on e.g. where it might be a good starting point.

Thanks a lot!

fatPeter commented 1 month ago

Thank you for your interest.

In mini-splatting, the formulation of render_depth does not support gradient backward. This is because, we formulate the rendered depth as $d=d{i{max}}$. Thus, only the selected Gaussian $G{i{max}}$ can be optimized if render_depth is differentiable. However, it is still possible to leverage render_depth as the pesudo depth.

Note that the forward and backward pass of depth rendering has already been updated into the original 3DGS repo (but is reverted). Check out this branch (NEW FEATURES) if you just want to know how to make depth differentiable.

yzhang-gh commented 1 month ago

Thank you for the nice explanation 😄.

I also happened to find that the original repo added depth regularization. Thanks for the information again.