Questions Regarding the Use of Diffusion Models for Reconstruction Tasks

[sIHURs]

Hi~~

Regarding the use of diffusion models to reconstruct objects or scenes, I have always a question. Compared to non-generative models or GANs, their reconstructions are deterministic. However, with diffusion models or VAEs, the sampling process from a distribution introduces significant bias and uncertainty in the reconstruction results, doesn’t it? Even if we add appropriate conditions, if the condition lacks information, wouldn't it lead to a reconstruction result that deviates significantly from the ground truth?

For example, in a simple case where like we are reconstructing a chair, but the incomplete point cloud only contains the upper part of the chair, and the lower part is entirely missing, the diffusion model might generate a chair with four legs or other various leg shapes, depending on the distribution in the diffusion model.

I would greatly appreciate it if you could share your thoughts about it.

yifan

[nuneslu]

Hi,

I guess that the main difference between generative models and non-generative models in this regard is that non-generative models build an interpolation/average representation of the object/scene. Generative models are trained to extrapolate the generation out of the average representation. This can be used as an advantage; in the example you mentioned about the chair, a non-generative approach would reconstruct the chair with four straight legs since this is the most common pattern. Generative models can try to extrapolate the shape of the legs based on the shape of the top part of the chair. For example, if the top part shows some curved patterns, it would try to replicate these patterns in the bottom part as well. I see that as the main advantage of using generative models in reconstruction tasks.