search

RaymondJiangkw / COGS

[SIGGRAPH'24] A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose
https://raymondjiangkw.github.io/cogs.github.io/
Other
68 stars 6 forks source link

[Question] How to backpropagate the gradient of the camera ? #9

Closed Master-cai closed 3 months ago

Master-cai commented 3 months ago

Hiya! Thanks for your wonderful work! I would like to ask some questions about the the gradient of the camera (or the viewpoint).

  1. I'm trying to backpropagate the gradient of the camera, when I use the vanilla diff-gaussian-rasterization and set the camera pose as nn.Parameter and use a adam to optimize them, I found the camera pose can not be update as they don't have grad. Does the vanilla rasterization can't backpropagate gradient of camreas? If so, could you kindly give me some advice about how to implement it ? I'm not very familiar with the CUDA programing.

  2. If I use your surface-rasterization, can I only backpropagate the gradient of the camera by setting other vars to requires_grad_(False) such as Fov? (It seems that your rasterization backpropagate grad of many vars).

Thank you very much! @RaymondJiangkw

RaymondJiangkw commented 3 months ago

Hi, thanks for your persistent interest. :)

The vanilla rasterizer does not support the differentiation through the camera parameters. You could either, refer to COLMAP-Free 3DGS for a work-around to differentiate through the camera parameters without modifying the CUDA, or use our provided custom rasterizer. And yes, you can only optimize the camera parameters while making other variables static by setting requires_grad_(False).