Some prolems with rot_4d when mapping

[Ma1hn]

I want to extend your 4D representation to slam.

when test tum datasets: freiburg3_walking_xyz I met the following problems:

1. in first time, I initilize the gaussians by first frame depth and color image
2. then I set the learning rate (ignoring the cam_params), to optimize guassians by descreasing rgb and depth loss ( means_xyz=0.0025, means_t=-1.0, shs=0.0025, opacities=0.05, scales=0.005, scales_t=0.005, rots=0, rots_r=0, cam_unnorm_rots=0, cam_trans=0, )
3. my question is when I set rots and rots_r =0, it can optimize a relatively good result 1st iteration rendered color image 30th iteration rendered color image
4. but when I set rots and rots_r as non_zero, like 0.0001, the rendered color becomes more and more dark with optimization goes on and finally is pure black image. 1st iteration rendered color image with rots 10th iteration rendered color image 30the iteration rendered color image ,

Do you ever meet some problems like this, and do you know any method to solve it?

[Alexander0Yang]

Do you mean setting the learning rate of rotations to 0? Since computeCov2DCUDA is also depend on the 3D means, the issue you mentioned in #48 would also result in incorrect gradients for rotations, which might cause problems in certain scenarios. Additionally, we’ve overridden the FOV to -1.0 because we found that this leads to better performance in specific cases, but it might also cause problems in your task. Anyway, I suggest to check there is any unusual behavior in computeCov2DCUDA.

[Ma1hn]

thanks for your reply. I was wondering if rigid loss and motion loss play a role in training gaussians params in unconstrained conditions, like sequence rgbd datasets.

[Alexander0Yang]

In our experience, rigid loss is very important for under-constrained conditions (although we have not used it in this work), but it usually requires careful tuning.

[Ma1hn]

Thanks!!!