overfitting problem

[fuyabo]

I experimented quite a lot and noticed that by reducing the number of training views from 50 to 5, (equal angular spaced), the quality of the synthesized novel view degraded significantly, though the synthesized view for the training angle is always almost perfect. It looks like the learnt 3D gaussian overfits to the training images, where the splatted 2D view is very similar to the training images. However, for new angles, it collapses, generating a lot of hallucinations. Would you please point to me what I might have done incorrectly, and any tips to get as good results as your paper indicated that from only 5 views, you can get excellent CT when reconstructed using newly synthesized projections. Really appreciate your response!

[caiyuanhao1998]

Hi, thanks for your interest,

Please do not reduce the training views.

The CT reconstruction in the paper is just an evaluation of the projections.

[fuyabo]

I think the writing in your paper is misleading then. In your paper, you said you created 95new projections from only 5 training views, and still able to recon decent CT.

On Sun, Sep 29, 2024 at 11:11 Yuanhao Cai @.***> wrote:

Hi, thanks for your interest,

Please do not reduce the training views.

The CT reconstruction in the paper is just an evaluation of the projections.

— Reply to this email directly, view it on GitHub https://github.com/caiyuanhao1998/X-Gaussian/issues/15#issuecomment-2381393334, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVILLMGX64BEQPKQVKVQLLZZAKCJAVCNFSM6AAAAABPBYJGSKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBRGM4TGMZTGQ . You are receiving this because you authored the thread.Message ID: @.***>

[caiyuanhao1998]

I did not say 5 training views. I said 5 original views. Look

And I never said we use 5 views to train the 3DGS, Look

The CT reconstruction is an evaluation of the rendered novel views. Take care.

[fuyabo]

I apologize if I misunderstood. Thank you for your response!

On Sun, Sep 29, 2024 at 11:58 Yuanhao Cai @.***> wrote:

I did not say 5 training views. I said 5 original views. Look 2024-09-29.11.55.07.png (view on web) https://github.com/user-attachments/assets/ec5e84d5-4d0e-4257-8876-3eda8174c8c0

And I never said we use 5 views to train the 3DGS, Look 2024-09-29.11.58.05.png (view on web) https://github.com/user-attachments/assets/db6abbb1-4e10-454c-bc67-95ddf3003409

The CT reconstruction is an evaluation of the rendered novel views. Take care.

— Reply to this email directly, view it on GitHub https://github.com/caiyuanhao1998/X-Gaussian/issues/15#issuecomment-2381408328, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVILLLSH3CKPQZZT2G3YXTZZAPUDAVCNFSM6AAAAABPBYJGSKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBRGQYDQMZSHA . You are receiving this because you authored the thread.Message ID: @.***>

[caiyuanhao1998]

No worries. I understand directly retrieving the radiodensity from 3DGS for CT reconstruction is very hard problem.

Our new NeurIPS work offers a solution for this:

https://arxiv.org/abs/2405.20693

Code for R2Gaussian will be released before the start date of NeurIPS 2024.

Please also have a look at my CVPR 2024 work SAX-NeRF:

https://github.com/caiyuanhao1998/SAX-NeRF

I develop this repo into a toolbox containing 9 algorithms for CT reconstruction. You may be interested :)