Why Translation normalization have a huge impact on the rendering result?

Miaosheng1 commented 3 months ago

Hi, I'm training the Mvsplat to reconstruct the street scene, and I find a question:

when i normalize the extrinsics (eg. translation ) using the following code, i can get a good result ( orange curve)。 scale_factor /= np.max(np.abs(wordl2camera[:, :3, 3])) wordl2camera[:, :3, 3] *= scale_factor
when i comment out the normalize code, the training PSNR has decreased significantly ( blue curve)。
The comparison curve of training process is as follows:

Can you provide some explanation for the phenomenon？

Normalize the translation Render Depth:

Unnormalize the translation Render Depth:

Corresponding Image:

donydchen commented 2 months ago

Hi @Miaosheng1, sorry for the late reply. I have been busy for the past few weeks.

Glad to see that you're trying to apply MVSplat to other datasets. It looks like the normalization operation will significantly affect the depth scale, leading to performance differences.

Below, I listed some suggestions that might help identify the main issues,

Do you train from scratch or fine-tune from the RE10K released weight? The depth range in KITTI (I assumed the dataset is KITTI?) is quite large. In this case, normalizing the translation might be helpful to align that with the RE10K dataset, leading to better performance if you are fine-tuning from the RE10K pre-trained model.
What are the near and far values in your settings?
What is the average scale of the scale_factor?
How is the quality of the depth predicted by the encoder instead of the Gaussian rendered ones? The encoder's predicted depth reflects the model more directly; hence, it is better for debugging. You can plot it by following https://github.com/donydchen/mvsplat/blob/6cf9deee0821b7ff10c33c0cb5e4189853467bcd/src/paper/generate_point_cloud_figure_mvsplat.py#L439-L448
How does it perform on the test set? Or how does it perform in the late training stage (I saw the curve ends at around 3K steps; how about 30K steps)?

With more details regarding the above-listed questions, we might be able to identify the main issues and figure out how to correctly configure the model in your dataset.

Miaosheng1 commented 1 month ago

Changing the near and far helps to improve the quality.

liucsg commented 1 month ago

Changing the near and far helps to improve the quality.

Could you tell how to apply MVSplat to kitti datasets?

donydchen / mvsplat