ruoshiliu
commented
4 months ago You also need to rotate the spherical harmonics of the Gaussians. Here's a thread discussing that: https://github.com/graphdeco-inria/gaussian-splatting/issues/176. But there's no conclusive answer afaik.
Hello,
Currently, I'm working on recovering camera poses and 3D reconstruction using Gaussian splattings. The approach involves using depth maps to unproject pixels for a specific frame, fitting them tightly (overfitting), and then determining the relative transformation to the next frame. My aim is to learn these transformations by adjusting parameters in the camera code, such as the world_to_view matrix. However, it appears that gradients will only propagate if I directly manipulate the rotation and translation of the Gaussian parameters themselves. Currently, I'm applying rotation and translation only to the xyz coordinates. Should I also apply them to the rotation parameters (rot params)? Q1: Is it possible for gradients to flow through the camera in the renderer, or is applying R+T to the params dirrectly the only possibility with the current version of diff_gaussian_rasterizer?
I've also conducted an experiment where I've learned an object and aim to render it using a rotation (R) and translation (T). When I apply these R+T transformations to the camera parameters and then render using 3DGS, everything seems fine. However, when I apply R+T to the xyz parameters of the Gaussians and use the default camera (Identity) with zero translation, parts of the Gaussians render strangely. Perhaps you can suggest what might be the problem? I've experimented with the camera_center parameter, and it seems to slightly influence the result.
Attached is an example: on the left is the result when I directly transform 3DGS, and on the right is when I transform camera parameters:
Here is an example - on left, when I transform 3DGS direcly, on right when I transform camera params:
I would appreciate any help, thanks in advance!