[Bug]: Distorted Outputs

[MrHorakhty]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you read FAQ on README?

• [X] I have updated WebUI and this extension to the latest version

What happened?

After a lot of tries I decided to use few words a prompts thinking maybe loras were conflicting but it seems like whenever I use animatediff output becomes distorted.

Steps to reproduce the problem

Write a prompt Activate only animatediff Generate Distorted outputs

What should have happened?

Non distorted outputs.

Commit where the problem happens

webui: extension:

What browsers do you use to access the UI ?

No response

Command Line Arguments

--no-half --no-progressbar-hiding --xformers

Console logs

No errors seems to be printed.

Startup time: 33.2s (prepare environment: 7.5s, import torch: 3.9s, import gradio: 1.0s, setup paths: 1.4s, initialize shared: 0.4s, other imports: 0.5s, load scripts: 4.4s, create ui: 5.5s, gradio launch: 3.4s, app_started_callback: 5.3s).2024-07-12 00:30:26,972 - AnimateDiff - INFO - AnimateDiff process start.
2024-07-12 00:30:26,973 - AnimateDiff - INFO - Loading motion module animatediff_lightning_4step_comfyui.safetensors from C:\Games\Stable Diffusion\stable-diffusion-webui\extensions\sd-webui-animatediff\model\animatediff_lightning_4step_comfyui.safetensors
2024-07-12 00:30:26,991 - AnimateDiff - INFO - Guessed animatediff_lightning_4step_comfyui.safetensors architecture: MotionModuleType.AnimateDiffV2
2024-07-12 00:30:29,355 - AnimateDiff - INFO - Injecting motion module animatediff_lightning_4step_comfyui.safetensors into SD1.5 UNet middle block.
2024-07-12 00:30:29,355 - AnimateDiff - INFO - Injecting motion module animatediff_lightning_4step_comfyui.safetensors into SD1.5 UNet input blocks.
2024-07-12 00:30:29,356 - AnimateDiff - INFO - Injecting motion module animatediff_lightning_4step_comfyui.safetensors into SD1.5 UNet output blocks.
2024-07-12 00:30:29,356 - AnimateDiff - INFO - Setting DDIM alpha.
2024-07-12 00:30:29,365 - AnimateDiff - INFO - Injection finished.
2024-07-12 00:30:29,365 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 3 frames.
  0%|                                                                                           | 0/20 [00:00<?, ?it/s]2024-07-12 00:30:29,453 - AnimateDiff - INFO - inner model forward hooked
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:21<00:00,  1.09s/it]
2024-07-12 00:30:52,134 - AnimateDiff - INFO - Restoring DDIM alpha.███████████████████| 20/20 [00:20<00:00,  1.08s/it]
2024-07-12 00:30:52,134 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet input blocks.
2024-07-12 00:30:52,135 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet output blocks.
2024-07-12 00:30:52,135 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet middle block.
2024-07-12 00:30:52,135 - AnimateDiff - INFO - Removal finished.
2024-07-12 00:30:52,136 - AnimateDiff - INFO - Saving output formats: GIF, PNG
2024-07-12 00:30:52,493 - AnimateDiff - INFO - AnimateDiff process end.
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:21<00:00,  1.08s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:21<00:00,  1.08s/it]

Additional information

Generation example

Specs GeForce RTX 4060 Ti 16GB Vram 32GB Ram

Also Pad Prompting is on and I tried changing Cross attention optimization to sdq but no luck. Edit: Tried with just control net and animmatediff active (only animatediff was enabled) same results.

[zappityzap]

Try using more frames.

[MrHorakhty]

Left it over night with 16 number of frames and batch size and 8 fps. Apparently power cut off in middle of it but png files were like these. edit: with prompt travel on by the way

[zappityzap]

Does it happen with other motion modules, like the ones from here: https://huggingface.co/conrevo/AnimateDiff-A1111/tree/main

Does it happen with other checkpoints? I can't see which one you're using there.

[MrHorakhty]

Normally I use mm_sd15_v3 and animatediff_lightning_4step_comfyui for motion modules and revanimated and dreamshaper for checkpoints. However after your message I tried mm_sd14_v1 and it seems to be fine. Thanks for the help.