search

omerbt / MultiDiffusion

Official Pytorch Implementation for "MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation" presenting "MultiDiffusion" (ICML 2023)
https://multidiffusion.github.io/
979 stars 57 forks source link

Equation 4 interpetation for the panorama use case #7

Closed kobybibas closed 1 year ago

kobybibas commented 1 year ago

Thanks for the awesome paper and very clear code!

For the panorama use case, can the method be reduced to the following implementation: At each de-noising step, take the average pixel value of the overlapping regions https://github.com/omerbt/MultiDiffusion/blob/f91b1c25fd14826d46f0f21c6f099b57fc446737/panorama.py#L138

If yes, how does the least squares formulation of the paper align with it? image

And again, thanks!

omerbt commented 1 year ago

Hi, thanks!

The panorama is a private case of the formulation in Eq. 4. Specifically, Fi map the panorama to square crops, and the weights Wi are set to one (Eq. 7 in the paper). A53B2712-60C6-4623-821E-F66EC68E78B7

In terms of implementation, indeed at each denoising step we take the average over overlapping regions. Note that the 'count' parameter amounts for the denominator in Eq. 5.

Let me know if you have any other questions:)