Img2Img for Video Generation

[MinSong2]

Model/Pipeline/Scheduler description

The proposed new community pipeline is stemmed from interpolate_stable_diffusion.py. Unlike interpolate_stable_diffusion, the proposed pipeline is interpolating between an initial image supplied by the user and an image generated by the user prompt.

At the following github site, https://github.com/OnomaAI/noci_diffusers/tree/main/img2video, I provided the code snippet for how to use the custom pipeline.

Open source status

• [X] The model implementation is available
• [x] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://github.com/OnomaAI/noci_diffusers/blob/main/img2video/img2img_video_stable_diffusion.py

[patrickvonplaten]

Hey @MinSong2,

Thanks for opening the issue. Note that we need model weights to be available in order to add an architecture.

[MinSong2]

Hello @patrickvonplaten,

Thank you for your reply! I used stabilityai/stable-diffusion-2 as model weights, which means that I did not update the weights. I might have misunderstood the "Open Source Status" option "The model weights are available (Only relevant if addition is not a scheduler)."! I checked the checkbox for the option!

Many thanks! Min

[kadirnar]

Are you going to add it to the diffusers library? @MinSong2

I'm also working on producing videos using the img2img pipeline. https://github.com/kadirnar/Custom-Diffusion