Reproducing the results of Dynamic3DGaussians on Neural 3D dataset.

[Ahmad-Jarrar]

You have reported the results of Dynamic3DGaussians for Neural 3D dataset. I am having trouble running the method on this dataset. Can you please provide the reformatting code/metadata file for Dynamic3DGaussians.

[lizhan17]

Hi Since the neural 3D video dataset does not provide foreground/background masks, we turn off the background segmentation loss in Dynamic3DGaussians and just set the last column of init_pt.cld.npz as all ones. We also disable the floor loss and set the log_scales in initialize_optimizer to 0.005.

[Ahmad-Jarrar]

Thank you for your quick reply.

I followed your recommendations but it still doesnt solve the issue I am facing, I use he point cloud creating using colmap data that was generated using the pre_n3d.py script.

For the camera parameters, I tried two different techniques,

1. I created the camera extrinsic matrices (w2c) from colmap as:

   qvec = image.qvec
   tvec = image.tvec
   rot = qvec2rotmat(qvec)
   w2c = np.eye(4)
   w2c[:3, :3] = rot
   w2c[:3, 3] = tvec

2. I also tried using the dataset reader for dynerf dataset which extracts the poses directly from poses_bounds.npy and then generated the w2c matrix as:

R,T = pose
W2C = np.eye(4)
W2C[:3, :3] = R
W2C[:3, 3] = T

However, in both methods, the training fails even without bg_loss and floor_loss. During training, the PSNR starts around -40 and goes to 7 where renders become black and train loss becomes nan.

The initial image rendered before training looks completely white which makes me suspect something is wrong with the camera parameters.

After a 1500 iterations, it looks like this:

If you can provide more guidance in how you converted the dataset to appropriate format, it will be very helpful.

[lizhan17]

I think our data points are enough for training dynamic 3dgs as we have same dependence which is 3dgs.