How to use AnimateLCM image-to-video models.

[xyxxmb]

The diffusers code?

[CyberTimon]

Yes - I'm also interested in that one! 😄

[JACKHAHA363]

Anyone figured this out?

[JohnnyStreet]

Consider the following as pseudocode since I am adapting it from my own and editing it here to share. There may be some mistakes.

First grab the classes you will need from the HF space @ https://huggingface.co/spaces/wangfuyun/AnimateLCM-SVD/tree/main

# NOTE: I am NOT using StableVideoDiffusionPipeline included with diffusers. The output was way worse.
# Also note that in this example I have renamed the files to animatelcm_scheduler.py and animatelcm_pipeline.py

from animatelcm_scheduler import AnimateLCMSVDStochasticIterativeScheduler
from animatelcm_pipeline import StableVideoDiffusionPipeline

Here is the rest, also adapted from their code.

noise_scheduler = AnimateLCMSVDStochasticIterativeScheduler(
    num_train_timesteps=40,
    sigma_min=0.002,
    sigma_max=700.0,
    sigma_data=1.0,
    s_noise=1.0,
    rho=7,
    clip_denoised=False,
)

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt-1-1",
    scheduler=noise_scheduler,
    use_safetensors=True,
    torch_dtype=self.dtype,
    variant="fp16",
)
pipe.to("cuda")
pipe.enable_model_cpu_offload()

path = huggingface_hub.hf_hub_download(
    "wangfuyun/AnimateLCM-SVD-xt", "AnimateLCM-SVD-xt-1.1.safetensors"
)

model_select(path)

def model_select(file_path):
    print("Loading model weights", file_path)
    pipe.unet.cpu()
    state_dict = {}
    with safe_open(file_path, framework="pt", device="cpu") as f:
        for key in f.keys():
            state_dict[key] = f.get_tensor(key)
    missing, unexpected = pipe.unet.load_state_dict(state_dict, strict=True)
    pipe.unet.cuda()
    del state_dict
    return