search

Kahsolt / stable-diffusion-webui-prompt-travel

Travel between prompts in the latent space to make pseudo-animation, extension script for AUTOMATIC1111/stable-diffusion-webui.
The Unlicense
248 stars 22 forks source link

Would it be possible to walk between latents from images instead of latents from prompts? #21

Closed enn-nafnlaus closed 10 months ago

enn-nafnlaus commented 1 year ago

This may be too broad of a change for prompt travel. But given how the script walks from one latent generated by processing a SD prompt to another latent generated by processing a different SD prompt, I'd think it just as possible to walk directly between latents derived from images as well (such as is done in img2img).

We can of course run the prompt travel script from img2img, but we can't transition between two different images, just two different prompts manipulating a single image. There's batch img2img, but that'll run prompt travel once for each image.

Kahsolt commented 1 year ago

Yes it's possible 😃, and that have been partially implemented the day before yesterday, i.e. the ControlNet Travel. It does not touch any prompt/text_condtion/CLIP related stuff, but interpolates on the controlnet image_condition (like canny, depth, etc..) and deep in to the unet latents.

Unet-only travels (with out Control conditions) might be later split out to yet another standalone script (omg, there'd be more and more messy xxx-travels...) However, this is just a beginning of exploration of unet. Wait and see ;)

enn-nafnlaus commented 1 year ago

That will be perfect when paired with ControlNet's new batch mode. The results of this should be spectacular!!!

https://github.com/Mikubill/sd-webui-controlnet/pull/683

Kahsolt commented 10 months ago

No further updates, close temporarily.

I think that ideas on pure UNet travel will gradually (or has already?) been taken place by new technics. Things like EbSynth-like methods, AnimeDiff is much more promising :)