[Bug]: It only create a mosaic-like GIF.

cucumber5252 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you read FAQ on README?

[X] I have updated WebUI and this extension to the latest version

What happened?

I use it on WebUI but it create only a mosaic-like image.

Steps to reproduce the problem

When I set "Settings"=>"AnimateDiff">"When you have --xformers in your command line args, you want AnimateDiff to" {to Optimize attention layers with xformers} (I'll call this case "case A"),

it occured CUDA error.

so when I changed it to {Optimize attention layers with sdp (torch >= 2.0.0 required)} or {Do not optimize attention layers} (I'll call this case "case B"), my WebUI just create only a mosaic-like GIF.

What should have happened?

it should create normal GIF.

Commit where the problem happens

webui: https://github.com/TheLastBen/fast-stable-diffusion extension: https://github.com/continue-revolution/sd-webui-animatediff

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

No

Console logs

=====case A=====
Error completing request
*** Arguments: ('task(9w1kbkzkv7m7zyu)', '(best quality, ultra high resolution),(masterpiece),1girl, (collarbone),earring, necklace, (beach),(full body), \n0: smile,\n10: closed eyes, open mouth\n15: open eyes, closed mouth', '', [], 20, 'DPM++ 2M Karras', 1, 1, 7, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x78b7923d4e20>, 0, False, '', 0.8, -1, False, -1, 0, 0, 0, <scripts.animatediff_ui.AnimateDiffProcess object at 0x78b79260ece0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x78b79260ed70>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x78b78a7aa2c0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x78b79260eda0>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, 50) {}
    Traceback (most recent call last):
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 732, in process_images
        res = process_images_inner(p)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-animatediff/scripts/animatediff_cn.py", line 119, in hacked_processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 867, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 1140, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 235, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_samplers_common.py", line 261, in launch_sampling
        return func()
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 235, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stablediffusion/src/k-diffusion/k_diffusion/sampling.py", line 594, in sample_dpmpp_2m
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-animatediff/scripts/animatediff_infv2v.py", line 277, in mm_cfg_forward
        x_out = mm_sd_forward(self, x_in, sigma_in, cond_in, image_cond_in, make_condition_dict) # hook
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-animatediff/scripts/animatediff_infv2v.py", line 187, in mm_sd_forward
        out = self.inner_model(
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stablediffusion/src/k-diffusion/k_diffusion/external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
      File "/content/gdrive/MyDrive/sd/stablediffusion/src/k-diffusion/k_diffusion/external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_hijack_utils.py", line 26, in __call__
        return self.__sub_func(self.__orig_func, *args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_hijack_unet.py", line 48, in apply_model
        return orig_func(self, x_noisy.to(devices.dtype_unet), t.to(devices.dtype_unet), cond, **kwargs).float()
      File "/content/gdrive/MyDrive/sd/stablediffusion/ldm/models/diffusion/ddpm.py", line 858, in apply_model
        x_recon = self.model(x_noisy, t, **cond)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stablediffusion/ldm/models/diffusion/ddpm.py", line 1329, in forward
        out = self.diffusion_model(x, t, context=cc)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_unet.py", line 91, in UNetModel_forward
        return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stablediffusion/ldm/modules/diffusionmodules/openaimodel.py", line 776, in forward
        h = module(h, emb, context)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stablediffusion/ldm/modules/diffusionmodules/openaimodel.py", line 86, in forward
        x = layer(x)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-animatediff/motion_module.py", line 107, in forward
        return self.temporal_transformer(input_tensor, encoder_hidden_states, attention_mask)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-animatediff/motion_module.py", line 173, in forward
        hidden_states = block(hidden_states, encoder_hidden_states=encoder_hidden_states, video_length=video_length)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-animatediff/motion_module.py", line 237, in forward
        hidden_states = attention_block(
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-animatediff/motion_module.py", line 596, in forward
        hidden_states = self._memory_efficient_attention(query, key, value, attention_mask, optimizer_name)
      File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-animatediff/motion_module.py", line 494, in _memory_efficient_attention
        hidden_states = xformers.ops.memory_efficient_attention(
      File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/__init__.py", line 223, in memory_efficient_attention
        return _memory_efficient_attention(
      File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/__init__.py", line 321, in _memory_efficient_attention
        return _memory_efficient_attention_forward(
      File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/__init__.py", line 341, in _memory_efficient_attention_forward
        out, *_ = op.apply(inp, needs_gradient=False)
      File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/cutlass.py", line 196, in apply
        return cls.apply_bmhk(inp, needs_gradient=needs_gradient)
      File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/cutlass.py", line 260, in apply_bmhk
        out, lse, rng_seed, rng_offset = cls.OPERATOR(
      File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 692, in __call__
        return self._op(*args, **kwargs or {})
    RuntimeError: CUDA error: invalid configuration argument
    CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
    For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
    Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

=====case B=====
None console error

Additional information

=====using parameter===== (best quality, ultra high resolution),(masterpiece),1girl, (collarbone),earring, necklace, (beach),(full body), , smile,, Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 3986890625, Size: 512x512, Model hash: 6ce0161689, Model: v1-5-pruned-emaonly, AnimateDiff: "enable: True, model: mm_sd_v15_v2.ckpt, video_length: 16, fps: 8, loop_number: 0, closed_loop: R-P, batch_size: 16, stride: 1, overlap: 4, interp: Off, interp_x: 10, mm_hash: 69ed0f5f", Version: v1.6.0

=====using checkpoint===== AOM3_orangemixs.safetensors [d124fc18f0]

continue-revolution commented 1 year ago

Attention optimization should not drastically change your output quality. Even if you choose not to optimize, you will get nothing in your case. Please improve your prompts.

cucumber5252 commented 1 year ago

Attention optimization should not drastically change your output quality. Even if you choose not to optimize, you will get nothing in your case. Please improve your prompts.

Thanks for your help but I have two more questions.

First Is it only compatible with checkpoints based on SD v1.5??

Also, please tell me why the "CUDA ERROR" occured when I use AnimateDiff with xformers and how can I solve it.

THANKS FOR YOUR HELP

continue-revolution commented 1 year ago

CUDA ERROR is because of a bug in xformers. It has nothing to do with AnimateDiff. See https://github.com/facebookresearch/xformers/issues/845

AnimateDiff is only available for SD1.5. SDXL has HotShotXL, whose layers are almost identical to AnimateDiff, but it does not mean that you can mix them together.

continue-revolution / sd-webui-animatediff