Hi, thanks for sharing your technique, it looks very promising!
I am seeing some issues with hard objects noticeably being transparent when they are not in models trained by other approaches*. For example the table in the MipNeRF360 "garden" scene appears transparent in a model trained by this approach but not in the original.
Viewed from the side (notice the wooden legs and the bottom of the table) - original first, R3DG second:
Viewed from the top (notice you can see-through to the floor as the camera pans):
*Actually I'm not sure if the original model (source) was trained with the original 3DGS code or with an approach like SparseGS, or what the hyper-parameters were. But it would still be helpful to understand if this is a failure mode with known remedies. For example, is it possible that I created the wrong depth/mask images or that I need to tweak a hyperparameter?
JiatengLiu
commented
4 months ago
Hi, thanks for sharing your technique, it looks very promising!
I am seeing some issues with hard objects noticeably being transparent when they are not in models trained by other approaches*. For example the table in the MipNeRF360 "garden" scene appears transparent in a model trained by this approach but not in the original.
Viewed from the side (notice the wooden legs and the bottom of the table) - original first, R3DG second:
Viewed from the top (notice you can see-through to the floor as the camera pans):
https://github.com/NJU-3DV/Relightable3DGaussian/assets/8591901/c1661095-1e12-4694-9a91-648b9054dcae
https://github.com/NJU-3DV/Relightable3DGaussian/assets/8591901/e613528f-b737-448a-8088-4ada78a80d47
*Actually I'm not sure if the original model (source) was trained with the original 3DGS code or with an approach like SparseGS, or what the hyper-parameters were. But it would still be helpful to understand if this is a failure mode with known remedies. For example, is it possible that I created the wrong depth/mask images or that I need to tweak a hyperparameter?