search

continue-revolution / sd-webui-animatediff

AnimateDiff for AUTOMATIC1111 Stable Diffusion WebUI
Other
3.09k stars 256 forks source link

[Bug]: Unable to use Prompt S/R Replace with AnimateDiff enabled #159

Closed advtech92 closed 8 months ago

advtech92 commented 1 year ago

Is there an existing issue for this?

Have you read FAQ on README?

What happened?

I am attempting to use Prompt S/R as well as AnimateDiff. It will error out and fail to make the gif.

Steps to reproduce the problem

  1. Create your prompt with the phrase you want replace, my case 0.99
  2. Enable AnimateDiff, Don't change any settings.
  3. Select X/Y/Z
  4. Select Prompt S/R for X
  5. Enter in the values of 0.99, 0.6, 0.4, 0.3, 0.1, 0
  6. Errors out after making the batch photo

What should have happened?

Generate the gif correctly

Commit where the problem happens

webui: 1.6.0 extension: https://github.com/continue-revolution/sd-webui-animatediff/commit/325bac5f72aa4639a8ce62c5d493dd6f26a811a8

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

set COMMANDLINE_ARGS=--listen --api --enable-insecure-extension-access --no-half-vae --xformers
set PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.9,max_split_size_mb:256

Console logs

generating image for xyz plot: RuntimeError
Traceback (most recent call last):
  File "E:\Downloads\stable-diffusion-webui\scripts\xyz_grid.py", line 698, in cell
    res = process_images(pc)
  File "E:\Downloads\stable-diffusion-webui\modules\processing.py", line 732, in process_images
    res = process_images_inner(p)
  File "E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_cn.py", line 63, in hacked_processing_process_images_hijack
    return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\modules\processing.py", line 867, in process_images_inner
    samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
  File "E:\Downloads\stable-diffusion-webui\modules\processing.py", line 1528, in sample
    samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
  File "E:\Downloads\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 188, in sample_img2img
    samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "E:\Downloads\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling
    return func()
  File "E:\Downloads\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 188, in <lambda>
    samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 553, in sample_dpmpp_sde
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 203, in mm_cfg_forward
    x_out[_context] = self.inner_model(x_in[_context], sigma_in[_context], cond=make_condition_dict(cond_in[_context], image_cond_in[_context]))
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "E:\Downloads\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "E:\Downloads\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\modules\sd_unet.py", line 91, in UNetModel_forward
    return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 802, in forward
    h = module(h, emb, context)
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_mm.py", line 86, in mm_tes_forward
    x = layer(x, context)
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\motion_module.py", line 86, in forward
    return self.temporal_transformer(input_tensor, encoder_hidden_states, attention_mask)
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\motion_module.py", line 143, in forward
    hidden_states = self.norm(hidden_states)
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\Downloads\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 459, in network_GroupNorm_forward
    return originals.GroupNorm_forward(self, input)
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\normalization.py", line 273, in forward
    return F.group_norm(
  File "E:\Downloads\stable-diffusion-webui\venv\lib\site-packages\torch\nn\functional.py", line 2530, in group_norm
    return torch.group_norm(input, num_groups, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: Expected weight to be a vector of size equal to the number of channels in input, but got weight of shape [1280] and input of shape [32, 2560, 12, 8]

*** Error completing request
*** Arguments: ('task(21fuokh7c7g0hf8)', 0, '((realistic)), ((detailed)), ((expressive details)), ((best quality)), ((masterpiece)), solo, [1boy, brown hair, blue eyes, blue shirt, jeans, brown shoes: 1girl, large breasts, blonde hair, long hair, ponytail, green eyes, smile, black halterneck shirt, blue leggings, knee boots:0.99]', '', [], <PIL.Image.Image image mode=RGBA size=512x768 at 0x19E854DE9E0>, None, None, None, None, None, None, 20, 'DPM++ SDE Karras', 4, 0, 1, 1, 1, 7, 1.5, 0.75, 0, 768, 512, 1, 0, 0, 32, 0, '', '', '', [], False, [], '', <gradio.routes.Request object at 0x0000019F0768E6E0>, 8, False, '', 0.8, -1, False, -1, 0, 0, 0, True, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'Euler a', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'inpaint_global_harmonious', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'Euler a', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'inpaint_global_harmonious', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', <scripts.animatediff_ui.AnimateDiffProcess object at 0x0000019E854DECB0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000019F0768EAD0>, False, '1:1,1:2,1:2', '0:0,0:0,0:1', '0.2,0.8,0.8', 20, '* `CFG Scale` should be 2 or lower.', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 7, '0.99, 0.6, 0.4, 0.3, 0.1, 0', [], 0, '', [], 0, '', [], False, True, True, False, 0, False, '<p style="margin-bottom:0.75em">Recommended settings: Use from inpaint tab, inpaint at full res ON, denoise < 0.5</p>', 'None', 30, 4, 0, 0, False, 'None', '<br>', 'None', 30, 4, 0, 0, 4, 0.4, True, 32, 7, [], True, True, True, 32.0, 120, <tempfile._TemporaryFileWrapper object at 0x0000019E854DCAF0>, 'gif', False, None, None, False, 50) {}
    Traceback (most recent call last):
      File "E:\Downloads\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "E:\Downloads\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "E:\Downloads\stable-diffusion-webui\modules\img2img.py", line 206, in img2img
        processed = modules.scripts.scripts_img2img.run(p, *args)
      File "E:\Downloads\stable-diffusion-webui\modules\scripts.py", line 601, in run
        processed = script.run(p, *script_args)
      File "E:\Downloads\stable-diffusion-webui\scripts\xyz_grid.py", line 739, in run
        processed = draw_xyz_grid(
      File "E:\Downloads\stable-diffusion-webui\scripts\xyz_grid.py", line 350, in draw_xyz_grid
        process_cell(x, y, z, ix, iy, iz)
      File "E:\Downloads\stable-diffusion-webui\scripts\xyz_grid.py", line 316, in process_cell
        cell_mode = processed_result.images[0].mode
    AttributeError: 'str' object has no attribute 'mode'

Additional information

Happens both with img2img and txt2img.

continue-revolution commented 1 year ago

This is because the processed result has been replaced by the path to gif. Unfortunately, I don’t have any idea on how to resolve it. The gif has been generated and saved, it is just not displayed on your UI.

advtech92 commented 1 year ago

Gotcha, I'll try it again. This message also pops up but I couldn't nab it the first time around: ** Error running before_process: E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py Traceback (most recent call last): File "E:\Downloads\stable-diffusion-webui\modules\scripts.py", line 611, in before_process script.before_process(p, script_args) File "E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 53, in before_process self.cn_hacker.hack(params) File "E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_cn.py", line 608, in hack self.hack_cn() File "E:\Downloads\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_cn.py", line 135, in hack_cn from scripts.controlmodel_ipadapter import (PlugableIPAdapter, ModuleNotFoundError: No module named 'scripts.controlmodel_ipadapter'

continue-revolution commented 1 year ago

Update your ControlNet will resolve this issue.

zappityzap commented 1 year ago

Appending here allows XYZ to draw the grid image. I'm not sure how useful it is to compare all the frames at once instead of videos side by side. It also means the individual frames show in the output window, making it hard to find the GIF, even when not using XYZ. Patching XYZ with an option to toggle drawing the grid would be better.

index 4b17d7f..0f58def 100644
--- a/scripts/animatediff_output.py
+++ b/scripts/animatediff_output.py
@@ -36,7 +36,7 @@ class AnimateDiffOutput:

         if len(video_paths) > 0:
             if not p.is_api:
-                res.images = video_paths
+                res.images.append(video_paths)
             else:
                 res.images = self._encode_video_to_b64(video_paths)
advtech92 commented 1 year ago

This is because the processed result has been replaced by the path to gif. Unfortunately, I don’t have any idea on how to resolve it. The gif has been generated and saved, it is just not displayed on your UI.

I have attempted it again, And it's only generating the first prompt of the XYZ Grid, not the other 5 in my case before it errors out. Here's a pastebin with the console: https://pastebin.com/KwPZLE17

zappityzap commented 1 year ago

I see all of my outputs with an XY plot, with and without CN+openpose and a source video. That console log doesn't have the same error about 'str' object has no attribute 'mode'. You may want to try updating A1111 and all extensions, and then restart A1111 completely.

advtech92 commented 1 year ago

I see all of my outputs with an XY plot, with and without CN+openpose and a source video. That console log doesn't have the same error about 'str' object has no attribute 'mode'. You may want to try updating A1111 and all extensions, and then restart A1111 completely.

I will double check that I am not missing an update. But, my A111 updates every it launches (edited in a git pull command) and as far as I can tell I am running the newest commit (5ef669de080814067961f28357256e8fe27544f4) of A111. I have since updated all my extensions as I updated it to remove said debug message that the last update caused. And I have also re-installed my ControlNet. I am trying to use S/R but I am not using a source video at all. I have also attempted without ControlNet and I still get the same result.

zappityzap commented 1 year ago

Could you share the simplest prompt and settings that produce the error, along with the entire console output from the launch of A1111?

The pastebin console log shows it crashing as soon as it starts sampling each image, and I can't reproduce that here.

advtech92 commented 1 year ago

Here's a console output from the very beginning: https://pastebin.com/VneicD4R And here's my prompt: ((realistic)), ((detailed)), ((expressive details)), ((best quality)), ((masterpiece)), solo, [1boy, brown hair, blue eyes, blue shirt, jeans, brown shoes: 1girl, large breasts, blonde hair, long hair, ponytail, green eyes, smile, black halterneck shirt, blue leggings, knee boots:0.99] Negative prompt is just (easynegative:1) I also have ADetailer running but does faces after the fact. It will do the first prompt but it won't do the rest of the S/R replacement which is 0.99, 0.6, 0.4, 0.3, 0.1 and 0. I also have to reboot A1111 after attempting to do it because it won't do a second run.

zappityzap commented 1 year ago

Thank you. The only difference I'm seeing is ADetailer. Could you try without ADetailer?

advtech92 commented 1 year ago

Tried it again without ADetailer enabled. Here's the console output: https://pastebin.com/x3jT2x8a I am still getting that str error

zappityzap commented 1 year ago

The str error is expected. From the log, it appears to have created each image. Can you confirm the images were saved? Does it continue to work on a second run?

advtech92 commented 1 year ago

The str error is expected. From the log, it appears to have created each image. Can you confirm the images were saved? Does it continue to work on a second run?

So, the grid only shows the first gif, I only see the first gif in the output, but it doesn't appear to process the rest of the XYZ script. And it doesn't let me attempt a second run after. It gives the weight error that appears above the str error.