About Paper

[GasaiYU]

Thanks for your excellent work!

I have a question about your paper.

At the rendering stage, what is your input? I understand the input are initial 3d gaussian point clouds and the timestamp t.

So your method is different from the input of the method HexPlane, whose input is some views' vidoes or dynamic images? To synthethis a novel view's dynamic scences.

[GasaiYU]

Can you tell me the difference of the input at the rendering stage between these 2 papers?

[guanjunwu]

Hi, thanks for your nice problem! Actually, the input of our whole pipeline is a camera pose $M=[R,T]$ with its intrinsic $K$ at time $t$. Then we can render the image $I'$ by our pipeline. $I' =Splatting(M;G')$, where $G'=F(G,t)$ is a deformed 3D Gaussians at timestamp $t$, which is calculated by a set of canonical 3D Gaussians $G$, timestamp $t$ and a our proposed Gaussian deformation network $F$. And $Splatting$ is differential Gaussian splatting algorithm as referred in 3D Gaussian Splatting. And we supervised whole model by ground truth image $I$ by the L1 Loss $L=|I-I'|$. HexPlane is a NeRF-based method, they rendering a image by $I' = VolumeRendering(M;c,\sigma)$, during volume rendering process, a ray is casted from camera to pixels and a set of points $x$ are sampled. Then a neural network including HexPlane and MLP is adopted to calculate the color and density of the points. Then volume rendering algorithm is used to blend the points from the rays. As a matter of the fact, both 4D Gaussians and HexPlane paper are used for the same task: novel view synthesis in dynamic scenes. And the technique is totally different. We are mainly based on 3D Gaussians, the HexPlane module is used to encode 3D Gaussians and we use the MLP to calcualte the deformation of each 3D Gaussians.The rendering process is mainly based on differential splatting algorithm. And the HexPlane paper is a extension of NeRF, the HexPlane module is to connect the spatial points, the rendering process is volume rendering. And the MLP in HexPlane paper is used to calculate the color and density.