Different results after cropping images

[brianneoberson]

Hello,

I was able to train your model on the DNA dataset and got quite good results. However, I tried cropping the DNA images since most of the image was taken up by background pixels and trained again with the same parameters but the results are a lot worse. Do you have an idea why this happens? Here is an example render of the same frame:

• with original data:

• with cropped data:

Here are the parameters that I used for both:

{
    "iterations": 30000,
    "test_iteration": 30000,
    "duration": 30,
    "resolution": 2,
    "test_views": [11, 25, 37, 57],
    "model": "ours_lite",
    "scaling_lr": 0.0015,
    "preprocesspoints": 3,
    "rdpip": "train_ours_lite",
    "loader": "colmap",
    "densify": 3,
    "desicnt": 6
  }

[lizhan17]

if you crop the images, the intrics is changed.

it seems the camera center is not correct. (points are mapping to positions with a small shift by different camera)

[brianneoberson]

Hi,

I made sure to update the instrinsics, namely the cx and cy parameters based on the bounding box I used for cropping, so I don't think that is the problem.

However, I realized that the way I was computing the bounding boxes for cropping would lead to different results at each timestamp. This means that the principal point would be different at each timestamp. I tried cropping the DNA dataset again using the same bounding boxes at each timestamp and trained on your code again and this time the results were good.

But this doesn't seem right, the principal points shouldn't necessarily have to be consistent over time, do they?

[lizhan17]

yes, not necessarily be consistent over time. ( likes cameras are moving overtime) But it is harder to have not consistent principal points. we optmized all the points jointly.

[brianneoberson]

Actually, I found that in https://github.com/oppo-us-research/SpacetimeGaussians/blob/3213738943392b9a0a96ece1ab40cf993021aaa8/thirdparty/gaussian_splatting/scene/dataset_readers.py#L79 you are first looping over the camera extrinsics and then over the frames, meaning the extrinsics and intrinsics of the first frame are being used for all subsequent frames. This isn't a problem if the cameras are fixed, but in a scene where the cameras move or the instrinsics change (like in my case after cropping the images) then this would lead to wrong results.