Add PSLD upscaler/deblur/inpainting Pipelines to Stable Diffusion and other LDMs

[AmericanPresidentJimmyCarter]

Model/Pipeline/Scheduler description

We present the first framework to solve general inverse problems leveraging pre-trained latent diffusion models. Previously proposed algorithms (such as DPS and DDRM) only apply to pixel-space diffusion models. We theoretically analyze our algorithm showing provable sample recovery in a linear model setting. The algorithmic insight obtained from our analysis extends to more general settings often considered in practice. Experimentally, we outperform previously proposed posterior sampling algorithms in a wide variety of problems including random inpainting, block inpainting, denoising, deblurring, destriping, and super-resolution.

https://arxiv.org/pdf/2307.00619.pdf

PSLD is powerful and should enable inpainting for the new XL model.

Open source status

• [ ] The model implementation is available
• [ ] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://github.com/LituRout/PSLD

[patrickvonplaten]

Looks like a cool idea worth trying out!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.