Travel between prompts in the latent space to make pseudo-animation, extension script for AUTOMATIC1111/stable-diffusion-webui.
Try interpolating on the hidden vectors of conditioning prompt to make seemingly-continuous image sequence, or let's say a pseudo-animation. 😀
Not only prompts! We also support non-prompt conditions, read => README_ext.md ~
⚠ 我们成立了插件反馈 QQ 群: 616795645 (赤狐屿),欢迎出建议、意见、报告bug等 (w
⚠ We have a QQ chat group (616795645) now, any suggestions, discussions and bug reports are highly wellllcome!!
ℹ 实话不说,我想有可能通过这个来做ppt童话绘本甚至本子……
ℹ 聪明的用法:先手工盲搜两张好看的图 (只有prompt差异),然后再尝试在其间 travel :lolipop:
⚠ Remeber to check "Always save all generated images" on in the settings tab, otherwise "upscaling" and "saving intermediate images" would not work. ⚠ 记得在设置页勾选 “总是保存所有生成的图片”,否则 上采样 与 保存中间图片 将无法工作。
⚪ Compatibility
The latest version v3.1
is synced & tested with:
v1.5.1
, tag v1.5.1v1.1.229
, commit eceeec7a7e⚪ Features
v3.1
supports SDXL v1.0 modelsv3.0
re-impl core with sd-webui v1.4.0
callbacks; this new implementation will be slower, but more compatible with other extensionsv2.7
add RIFE to controlnet-travel, skip fusion (experimental)v2.6
add a tkinter GUI for postprocess toolchainv2.5
add controlnet-travel script (experimental), interpolating between hint conditions instead of prompts, thx for the code base from sd-webui-controlnet v2.3
integrate basic function of depth-image-io for depth2img modelsv2.2
add 'slerp' linear interpolation methodv2.1
add experimental 'replace' mode again, it's not smooth interpolationv2.0
add optional external post-processing pipeline to highly boost up smoothness, greate thx to Real-ESRGAN and RIFE!!v1.5
add upscale options (issue #12); add 'embryo' genesis, reproducing idea of stable-diffusion-animation except FILM support (issue #11)v1.4
remove 'replace' & 'grad' mode support, due to webui's code changev1.3
work in a more 'successive' way, idea borrowed from deforum ('genesis' option)v1.2
walk by substituting token embedding ('replace' mode)v1.1
walk by optimizing condition ('grad' mode)v1.0
interpolate linearly on condition/uncondition ('linear' mode)⚪ Fixups
v1.1.424
v1.1.229
v1.1.116
v2.4
bug fixes on script hook, now working correctly with extra networks & sd-webui-controlnetImportError: cannot import name 'single_sample_to_image'
)v1.5
apply zero padding when condition length mismatch (issue #10: RuntimeError: The size of tensor a (77) must match the size of tensor b (154) at non-singleton dimension 0
), typo in demo filenamev1.4
keep up with webui's updates (issue #9: AttributeError: 'FrozenCLIPEmbedderWithCustomWords' object has no attribute 'process_text'
)#bdd8bed
fixup no working when negative prompt is left empty (issue #6: neg_prompts[-1] IndexError: List index out of range
)v1.2-fix2
keep up with webui's updates (error ImportError: FrozenCLIPEmbedderWithCustomWords
)v1.2-fix1
keep up with webui's updates (error AttributeError: p.all_negative_prompts[0]
)⚠ this script will NOT probably support the schedule syntax (i.e.: [prompt:prompt:number]
), because interpolate on mutable conditions requires sampler level tracing which is hard to maintain :(
⚠ this script will NOT probably work together with hires.fix
due to some inner conceptual/logical conflict of denoising_strength
, you can alternatively perform batch-upscale then batch-img2img.
steps
, sampler
, cfg factor
, seed
, etc.)seed
will be forcely fixed through all processes, you can still set subseed = -1
to allow more variances⚪ Txt2Img
sampler \ genesis | fixed | successive | embryo |
---|---|---|---|
Eular a | |||
DDIM |
⚪ Img2Img
sampler \ genesis | fixed | successive | embryo |
---|---|---|---|
Eular a | |||
DDIM |
post-processing pipeline (case i2i-f-ddim
):
w/o. post-processing | w/. post-processing |
---|---|
other stuff:
reference image for img2img | embryo image decoded case i2i-e-euler_a with embryo_step=8 |
---|---|
⚪ ControlNet support
prompt-travel with ControlNet (depth) | controlnet-travel (depth) |
---|---|
Example above run configure:
Prompt:
(((masterpiece))), highres, ((boy)), child, cat ears, white hair, red eyes, yellow bell, red cloak, barefoot, angel, [flying], egyptian
((masterpiece)), highres, ((girl)), loli, cat ears, light blue hair, red eyes, magical wand, barefoot, [running]
Negative prompt:
(((nsfw))), ugly,duplicate,morbid,mutilated,tranny,trans,trannsexual,mutation,deformed,long neck,bad anatomy,bad proportions,extra arms,extra legs, disfigured,more than 2 nipples,malformed,mutated,hermaphrodite,out of frame,extra limbs,missing arms,missing legs,poorly drawn hands,poorty drawn face,mutation,poorly drawn,long body,multiple breasts,cloned face,gross proportions, mutated hands,bad hands,bad feet,long neck,missing limb,malformed limbs,malformed hands,fused fingers,too many fingers,extra fingers,missing fingers,extra digit,fewer digits,mutated hands and fingers,lowres,text,error,cropped,worst quality,low quality,normal quality,jpeg artifacts,signature,watermark,username,blurry,text font ufemale focus, poorly drawn, deformed, poorly drawn face, (extra leg:1.3), (extra fingers:1.2),out of frame
Steps: 15
CFG scale: 7
Clip skip: 1
Seed: 114514
Size: 512 x 512
Model hash: animefull-final-pruned.ckpt
Hypernet: (this is my secret :)
linear
: linear interpolation on condition/uncondition of CLIP outputreplace
: gradually replace of CLIP outputtoken
: per token-wise vectorchannel
: per channel-wise vectorrandom
: per point-wise elementsimiliar
: from the most similiar first (L1 distance)different
: from the most different firstrandom
: just randomlyembryo
: pre-denoise few steps, then hatch a set of image from the common embryo by linear interpolationlen(stages)-1
, separate by comma, e.g.: 12, 24, 36
fixed
: starts from pure noise in txt2img pipeline, or from the same ref-image given in img2img pipelinesuccessive
: starts from the last generated image (this will force txt2img turn to actually be img2img from the 2nd frame on)embryo
: starts from the same half-denoised image, see => How does it work? successive
)embryo
)0
to disable file saving::2
to get only even frames, set :-1
to drop last frame)Easiest way to install it is to:
Manual install:
There are still two steps away from a really smooth and high resolution animation, namely image super-resolution & video frame interpolation (see third-party tools
below).
⚠ Media data processing is intrinsic resource-exhausting, and it's also not webui's work or duty, hence we separated it out. 😃
⚪ auto install (Windows)
cd tools & install.cmd
Access denied.
, try run it again until you see Done!
without errors 😂curl schannel ... Unknown error ... certificate.
, the downloader not work due to some SSL security reasons, just turn to install manually...⚪ manually install (Windows/Linux/Mac)
ℹ Understand the tools
folder layout first => tools/README.txt
ℹ If you indeed wanna put the tools elsewhere, modify paths in tools/link.cmd and run cd tools & link.cmd
😉
For Windows:
realesrgan-ncnn-vulkan-20220424-windows.zip
)
realesr-animevideov3.pth
)rife-ncnn-vulkan-20221029-windows.zip
)ffmpeg-release-full-shared.7z
or ffmpeg-git-full.7z
)For Linux/Mac:
tools
folder layout, manually apply chmod 755
to the executablesffmpeg
can be easily found in your app store or package manager, run like apt install ffmpeg
; DO NOT need to link it under tools
folder⚪ tkinter GUI (Windows/Linux/Mac)
For Windows:
manager.cmd
, to start webui's python venvinstall
(only setup once)run
For Linux/Mac:
../../venv/Scripts/activate
, to start webui's python venvpip install -r requirements.txt
(only setup once)python manager.py
ℹ find usage help message in right click pop menu~
⚪ cmd script (Windows) - deprecated
postprocess.cmd path/to/<image_folder>
from command linepostprocess.cmd
iconsynth.mp4
⚪ extensions that inspired this repo
⚪ third-party tools
⚪ my other experimental toy extensions
by Armit 2022/11/10