An example about animatediff.cli functions

[darkcurrent]

Hi @s9roll7 , I am trying to use your library in my python script in the following way. In this way, the controlnet_openpose and controlnet_softedge folders appear as expected in the 00_detectmap folder as specified in the config file. So the first stage is working. But the code goes to the next step I get the following error. If you can prepare a small example of using your library in a python project as a package I would appreciate it.

I would like to be able to run commands like generate, tile_upscale, refine that we use via CLI in a python environment. I'm sure I'm making a simple mistake, I'd appreciate if you can help me.

from animatediff import cli
from pathlib import Path
cli.generate(
             config_path=Path("config/prompts/prompt_travel.json"), 
             width=384, height=256, length=64, context=16, overlap=16, stride=16, 
             repeats=1, device="cuda", use_xformers=True, force_half_vae=False, 
             out_dir=Path("output/"), no_frames=False, save_merged=True, version=True
)

# OUTPUT

INFO     Pipeline already loaded, skipping initialization                                       [cli.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py):[348](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py#348)
         INFO     No TI embeddings found                                                                  [ti.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/pipelines/ti.py):[102](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/pipelines/ti.py#102)
         INFO     loading c='controlnet_openpose' model                                             [generate.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/generate.py):[482](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/generate.py#482)
         INFO     loading c='controlnet_softedge' model                                             [generate.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/generate.py):[482](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/generate.py#482)
02:31:48 INFO     Sending pipeline to device "cuda"                                                  [pipeline.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/utils/pipeline.py):[22](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/utils/pipeline.py#22)
         INFO     Selected data types: unet_dtype=torch.float16, tenc_dtype=torch.float16,             [device.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/utils/device.py):[90](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/utils/device.py#90)
                  vae_dtype=torch.bfloat16                                                                         
         INFO     Using channels_last memory format for UNet and VAE                                  [device.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/utils/device.py):[111](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/utils/device.py#111)
         INFO     Saving prompt config to output directory                                               [cli.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py):[378](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py#378)
         INFO     Initialization complete!                                                               [cli.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py):[386](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py#386)
         INFO     Generating 1 animations                                                                [cli.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py):[387](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py#387)
         INFO     Running generation 1 of 1                                                              [cli.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py):[397](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py#397)
         INFO     Generation seed: 341774366206100                                                       [cli.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py):[403](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/cli.py#403)
         INFO     len( region_condi_list )=1                                                       [generate.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/generate.py):[1179](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/generate.py#1179)
         INFO     len( region_list )=1                                                             [generate.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/generate.py):[1180](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/generate.py#1180)
         INFO     multi_uncond_mode=False                                                         [animation.py](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/pipelines/animation.py):[2386](file:///D:/Projects_ArGE/project1/animatediff-cli-prompt-travel/src/animatediff/pipelines/animation.py#2386)
---------------------------------------------------------------------------
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
d:\Projects_ArGE\project1\animatediff-cli-prompt-travel\handling_cli_script.ipynb Cell 10 line 1
----> 1 cli.generate(
      2              config_path=Path("config/prompts/prompt_travel.json"), 
      3              width=384, height=256, length=64, context=16, overlap=16, stride=16, 
      4              repeats=1, device="cuda", use_xformers=True, force_half_vae=False, 
      5              out_dir=Path("output/"), no_frames=False, save_merged=True, version=True
      6 )

File D:\Projects_ArGE\project1\animatediff-cli-prompt-travel\src\animatediff\cli.py:406, in generate(model_name_or_path, config_path, width, height, length, context, overlap, stride, repeats, device, use_xformers, force_half_vae, out_dir, no_frames, save_merged, version)
    401 seed = model_config.seed[idx % num_seeds]
    403 logger.info(f"Generation seed: {seed}")
--> 406 output = run_inference(
    407     pipeline=g_pipeline,
    408     n_prompt=n_prompt,
    409     seed=seed,
    410     steps=model_config.steps,
    411     guidance_scale=model_config.guidance_scale,
    412     unet_batch_size=model_config.unet_batch_size,
    413     width=width,
    414     height=height,
    415     duration=length,
    416     idx=gen_num,
    417     out_dir=save_dir,
    418     context_frames=context,
    419     context_overlap=overlap,
    420     context_stride=stride,
    421     clip_skip=model_config.clip_skip,
    422     controlnet_map=model_config.controlnet_map,
    423     controlnet_image_map=controlnet_image_map,
    424     controlnet_type_map=controlnet_type_map,
    425     controlnet_ref_map=controlnet_ref_map,
    426     no_frames=no_frames,
    427     img2img_map=img2img_map,
    428     ip_adapter_config_map=ip_adapter_config_map,
    429     region_list=region_list,
    430     region_condi_list=region_condi_list,
    431     output_map = model_config.output,
    432     is_single_prompt_mode=model_config.is_single_prompt_mode
    433 )
    434 outputs.append(output)
    435 torch.cuda.empty_cache()

File D:\Projects_ArGE\project1\animatediff-cli-prompt-travel\src\animatediff\generate.py:1182, in run_inference(pipeline, n_prompt, seed, steps, guidance_scale, unet_batch_size, width, height, duration, idx, out_dir, context_frames, context_stride, context_overlap, context_schedule, clip_skip, controlnet_map, controlnet_image_map, controlnet_type_map, controlnet_ref_map, no_frames, img2img_map, ip_adapter_config_map, region_list, region_condi_list, output_map, is_single_prompt_mode)
   1179 logger.info(f"{len( region_condi_list )=}")
   1180 logger.info(f"{len( region_list )=}")
-> 1182 pipeline_output = pipeline(
   1183     negative_prompt=n_prompt,
   1184     num_inference_steps=steps,
   1185     guidance_scale=guidance_scale,
   1186     unet_batch_size=unet_batch_size,
   1187     width=width,
   1188     height=height,
   1189     video_length=duration,
   1190     return_dict=False,
   1191     context_frames=context_frames,
   1192     context_stride=context_stride + 1,
   1193     context_overlap=context_overlap,
   1194     context_schedule=context_schedule,
   1195     clip_skip=clip_skip,
   1196     controlnet_type_map=controlnet_type_map,
   1197     controlnet_image_map=controlnet_image_map,
   1198     controlnet_ref_map=controlnet_ref_map,
   1199     controlnet_max_samples_on_vram=controlnet_map["max_samples_on_vram"] if "max_samples_on_vram" in controlnet_map else 999,
   1200     controlnet_max_models_on_vram=controlnet_map["max_models_on_vram"] if "max_models_on_vram" in controlnet_map else 99,
   1201     controlnet_is_loop = controlnet_map["is_loop"] if "is_loop" in controlnet_map else True,
   1202     img2img_map=img2img_map,
   1203     ip_adapter_config_map=ip_adapter_config_map,
   1204     region_list=region_list,
   1205     region_condi_list=region_condi_list,
   1206     interpolation_factor=1,
   1207     is_single_prompt_mode=is_single_prompt_mode,
   1208     callback=callback,
   1209     callback_steps=output_map.get("preview_steps"),
   1210 )
   1211 logger.info("Generation complete, saving...")
   1213 save_fn(pipeline_output, out_file=out_file)

File c:\Users\cagin\anaconda3\envs\env1\lib\site-packages\torch\utils\_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File D:\Projects_ArGE\project1\animatediff-cli-prompt-travel\src\animatediff\pipelines\animation.py:2656, in AnimationPipeline.__call__(***failed resolving arguments***)
   2654 # 6.5 - Infinite context loop shenanigans
   2655 context_scheduler = get_context_scheduler(context_schedule)
-> 2656 total_steps = get_total_steps(
   2657     context_scheduler,
   2658     timesteps,
   2659     num_inference_steps,
   2660     latents.shape[2],
   2661     context_frames,
   2662     context_stride,
   2663     context_overlap,
   2664 )
   2666 # 7. Denoising loop
   2667 num_warmup_steps = len(timesteps) - num_inference_steps * self.scheduler.order

File D:\Projects_ArGE\project1\animatediff-cli-prompt-travel\src\animatediff\pipelines\context.py:59, in get_total_steps(scheduler, timesteps, num_steps, num_frames, context_size, context_stride, context_overlap, closed_loop)
     49 def get_total_steps(
     50     scheduler,
     51     timesteps: list[int],
   (...)
     57     closed_loop: bool = True,
     58 ):
---> 59     return sum(
     60         len(
     61             list(
     62                 scheduler(
     63                     i,
     64                     num_steps,
     65                     num_frames,
     66                     context_size,
     67                     context_stride,
     68                     context_overlap,
     69                 )
     70             )
     71         )
     72         for i in range(len(timesteps))
     73     )

File D:\Projects_ArGE\project1\animatediff-cli-prompt-travel\src\animatediff\pipelines\context.py:61, in <genexpr>(.0)
     49 def get_total_steps(
     50     scheduler,
     51     timesteps: list[int],
   (...)
     57     closed_loop: bool = True,
     58 ):
     59     return sum(
     60         len(
---> 61             list(
     62                 scheduler(
     63                     i,
     64                     num_steps,
     65                     num_frames,
     66                     context_size,
     67                     context_stride,
     68                     context_overlap,
     69                 )
     70             )
     71         )
     72         for i in range(len(timesteps))
     73     )

File D:\Projects_ArGE\project1\animatediff-cli-prompt-travel\src\animatediff\pipelines\context.py:33, in uniform(step, num_steps, num_frames, context_size, context_stride, context_overlap, closed_loop)
     31 for context_step in 1 << np.arange(context_stride):
     32     pad = int(round(num_frames * ordered_halving(step)))
---> 33     for j in range(
     34         int(ordered_halving(step) * context_step) + pad,
     35         num_frames + pad + (0 if closed_loop else -context_overlap),
     36         (context_size * context_step - context_overlap),
     37     ):
     38         yield [e % num_frames for e in range(j, j + context_size * context_step, context_step)]

ValueError: range() arg 3 must not be zero

[hylarucoder]

overlap: Number of frames to overlap in context (default: context//4)

PS: When an error occurs, you can also print the correct command-line interface (CLI) parameters for troubleshooting purposes.

[darkcurrent]

Thank you,

I have manage to run generate and tile_upscale functions. But I have two more questions for you guys...

1. Can we use the rife interpolate command from separate python interpreter as well?

2. When the stride argument from the generate function is setted other than "0" i gave some errors in the separate python interpreter mode.

Using library on external python interpreter setting stride to 1 or more than 1 it gives this error:

Traceback (most recent call last):
  File "/home/cagin/story-bot/storybot/worker-services/worker_animatediff.py", line 203, in <module>
    out_dir_gen = generate(
  File "/home/cagin/.cache/animatediff-cli-prompt-travel/src/animatediff/cli.py", line 406, in generate
    output = run_inference(
  File "/home/cagin/.cache/animatediff-cli-prompt-travel/src/animatediff/generate.py", line 1182, in run_inference
    pipeline_output = pipeline(
  File "/home/cagin/miniconda3/envs/test/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/cagin/.cache/animatediff-cli-prompt-travel/src/animatediff/pipelines/animation.py", line 2896, in __call__
    down_block_res_samples,mid_block_res_sample = get_controlnet_result(context)
  File "/home/cagin/.cache/animatediff-cli-prompt-travel/src/animatediff/pipelines/animation.py", line 2700, in get_controlnet_result
    first_down = list(list(controlnet_result.values())[0].values())[0][0]
IndexError: list index out of range

Using directly the terminal it won't give any errors when -S argument is setted to "1". animatediff generate -c config/prompts/prompt_travel_test.json -W 256 -H 384 -L 64 -C 16 -S 1

Again trying with -S is setted to 2 it gives this error animatediff generate -c config/prompts/prompt_travel_test.json -W 256 -H 384 -L 64 -C 16 -S 2 Error:

19:41:11 INFO     No TI embeddings found                                                                                                                                      ti.py:102
         INFO     loading c='controlnet_openpose' model                                                                                                                 generate.py:482
         INFO     loading c='controlnet_softedge' model                                                                                                                 generate.py:482
         INFO     Sending pipeline to device "cuda"                                                                                                                      pipeline.py:22
         INFO     Selected data types: unet_dtype=torch.float16, tenc_dtype=torch.float16, vae_dtype=torch.bfloat16                                                        device.py:90
         INFO     Using channels_last memory format for UNet and VAE                                                                                                      device.py:111
Preprocessing images (ip_adapter)   0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2  [ 0:00:00 < -:--:-- , ? it/s ]
Saving Preprocessed images (ip_adapter)   0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/4  [ 0:00:00 < -:--:-- , ? it/s ]
19:41:12 INFO     Saving prompt config to output directory                                                                                                                   cli.py:378
         INFO     Initialization complete!                                                                                                                                   cli.py:386
         INFO     Generating 1 animations                                                                                                                                    cli.py:387
         INFO     Running generation 1 of 1                                                                                                                                  cli.py:397
         INFO     Generation seed: 341774366206100                                                                                                                           cli.py:403
         INFO     len( region_condi_list )=1                                                                                                                           generate.py:1179
         INFO     len( region_list )=1                                                                                                                                 generate.py:1180
         INFO     multi_uncond_mode=False                                                                                                                             animation.py:2386
  19% ━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42/220  [ 0:00:15 < 0:01:05 , 3 it/s ]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/cagin/.cache/animatediff-cli-prompt-travel/src/animatediff/cli.py:406 in generate          │
│                                                                                                  │
│    403 │   │   │   logger.info(f"Generation seed: {seed}")                                       │
│    404 │   │   │                                                                                 │
│    405 │   │   │                                                                                 │
│ ❱  406 │   │   │   output = run_inference(                                                       │
│    407 │   │   │   │   pipeline=g_pipeline,                                                      │
│    408 │   │   │   │   n_prompt=n_prompt,                                                        │
│    409 │   │   │   │   seed=seed,                                                                │
│                                                                                                  │
│ /home/cagin/.cache/animatediff-cli-prompt-travel/src/animatediff/generate.py:1182 in             │
│ run_inference                                                                                    │
│                                                                                                  │
│   1179 │   logger.info(f"{len( region_condi_list )=}")                                           │
│   1180 │   logger.info(f"{len( region_list )=}")                                                 │
│   1181 │                                                                                         │
│ ❱ 1182 │   pipeline_output = pipeline(                                                           │
│   1183 │   │   negative_prompt=n_prompt,                                                         │
│   1184 │   │   num_inference_steps=steps,                                                        │
│   1185 │   │   guidance_scale=guidance_scale,                                                    │
│                                                                                                  │
│ /home/cagin/miniconda3/envs/test/lib/python3.10/site-packages/torch/utils/_contextlib.py:115 in  │
│ decorate_context                                                                                 │
│                                                                                                  │
│   112 │   @functools.wraps(func)                                                                 │
│   113 │   def decorate_context(*args, **kwargs):                                                 │
│   114 │   │   with ctx_factory():                                                                │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                   │
│   116 │                                                                                          │
│   117 │   return decorate_context                                                                │
│   118                                                                                            │
│                                                                                                  │
│ /home/cagin/.cache/animatediff-cli-prompt-travel/src/animatediff/pipelines/animation.py:2896 in  │
│ __call__                                                                                         │
│                                                                                                  │
│   2893 │   │   │   │   │                                                                         │
│   2894 │   │   │   │   │   cur_prompt = prompt_encoder.get_current_prompt_embeds(context, laten  │
│   2895 │   │   │   │   │                                                                         │
│ ❱ 2896 │   │   │   │   │   down_block_res_samples,mid_block_res_sample = get_controlnet_result(  │
│   2897 │   │   │   │   │                                                                         │
│   2898 │   │   │   │   │   if c_ref_enable:                                                      │
│   2899 │   │   │   │   │   │   # ref only part                                                   │
│                                                                                                  │
│ /home/cagin/.cache/animatediff-cli-prompt-travel/src/animatediff/pipelines/animation.py:2700 in  │
│ get_controlnet_result                                                                            │
│                                                                                                  │
│   2697 │   │   │   │   │                                                                         │
│   2698 │   │   │   │   │   _down_block_res_samples=[]                                            │
│   2699 │   │   │   │   │                                                                         │
│ ❱ 2700 │   │   │   │   │   first_down = list(list(controlnet_result.values())[0].values())[0][0  │
│   2701 │   │   │   │   │   first_mid = list(list(controlnet_result.values())[0].values())[0][1]  │
│   2702 │   │   │   │   │   for ii in range(len(first_down)):                                     │
│   2703 │   │   │   │   │   │   _down_block_res_samples.append(                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: list index out of range

it looks like the same error but it occurs only when -S is setted to 2 while separate python interpreter option gives this error when the stride argument is setted even 1...