A technial clarification

[GopiRajuMatta]

Hello @ethliup!

Thank you for nice work and many congratulations..

In this framework, what ensures that each image should be sharp and it's a deblurred version of input.. It might happen set of rendered images(which are blurry), when we average them can still ensure output will be as blurry as input..

Can you please clarify on this.

Thank you Gopi

[LingzheZhao]

Hi, I also noticed that you asked the same question in BeNeRF. As mentioned before, multi-view consistency of NeRF/3DGS can do this job.

Although all images are blurry, they are captured at different locations and have different blur patterns, thus they can provide enough information to recover the 3D sharp scene (compensate each other images' blurriness) during training through our differentiable physical motion-blur imaging (rendering) model.

[LingzheZhao]

There would be some failure cases:

1. Bad initialization: COLMAP failed to estimate camera pose and point cloud from the severely blurred images, making the problem very difficult to optimize to the convergence;
2. Too few images: cannot provide enough information to recover geometry and sharp details;
3. Extreme motion blur / resolution too high: You would need a much larger num_virtual_views to synthesize the blur image. In this case, you may find some difference between the synthesized blur image and the input blur image after training. You can instead downscale the images to make it easier to train;
4. Overfitting: You would see some high-frequency artifacts in the rendered images. This could happen when you add too many Gaussians in the scene and they become floaters. However, if you check the synthesized blur image, you would find it very similar to the input. Theoretically the deblurring problem (ill-posed inverse problem) could have infinite solutions, and the one with high-frequency artifacts can be one of the solutions to synthesize the blur image. But the Occam's razor principle usually holds in real life, thus the real solution would be very likely to be the simplest one, i.e. correct geometry, textures and camera pose with less number of Gaussians.

[GopiRajuMatta]

Thank you @LingzheZhao, very helpful!

[GopiRajuMatta]

If time permits, I want to meet online and discuss few things with you, it will be really helpful! Please let me know.

Thank you Gopi

[akawincent]

If time permits, I want to meet online and discuss few things with you, it will be really helpful! Please let me know.

Thank you Gopi

Hi @GopiRajuMatta

Sure, We can arrange an online meeting. https://github.com/WU-CVGL/BeNeRF/issues/4#issuecomment-2470192843