Is LCM lora compatible ? because I get "could not patch. key doesn't exist in model: diffusion_model.output_blocks.2.1.conv.weight"

[Pimax1]

HI, LCM is so great but I can't it to work with AnimateDiff Loader.

as soon as I plug the lcm in ANimateDiff Loader and try to run it, I get this message: could not patch. key doesn't exist in model: diffusion_model.output_blocks.2.1.conv.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.5.2.conv.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.8.2.conv.weight

and in the end LCM is not applied. My comfyui is updated and I have latest versions of all custom nodes. everything works great except for LCM + AnimateDiff Loader.

Someone got it to work ? LCM can speed up inference time by a factor 10, its really a must have now.

help please !

[Pimax1]

coming back on this issue, It seems to work (generates something much faster than other samplers) but with animatediff loader, the results will be very poor, seriously lacking details. I saw some people online using this LCM lora with animateDiff loader too, and not realising some weights are not patched (they have the same "could not patch" message in their console). they tend to use it anyway with higher than necessary steps to compensate, but that wont solve the quality issue in the end.

So it gives you the illusion it is somehow working, but the quality is so seriously degraded compared to what this lora should produce (just unplug AnimateDiff loader and you get good quality).

I am not expert but I think the fact that it cant patch some weights when used with animatediff prevents generating good results.

I hope someone can help

[Kosinkadink]

Gotcha, today I will look into seeing if I can move the unet motion module injection to happen after all unet patches are applied, and unet motion module ejection to happen before all unet patches are attempted to be unapplied. This should fix some potentially hidden bugs up to this point. I'll keep you posted.

[Kosinkadink]

Currently (not sure if there is anything i can do about this even after i see if there is a problem), v2-based motion modules don't work well with LCM, so I'd try LCM with v1-based motion modules.

[Nebuluss]

I was wondering as well ! thanks for the answer

[Kosinkadink]

I added a new branch called new_injection that should in theory fix the issue, assuming that this is something that happens only when applied with AD and doesn't happen with vanilla SD. To test it out, you can cd into custom-nodes/ComfyUI-AnimateDiff-Evolved, and do git checkout new_injection (note, you will likely need to do a git pull first so your local instance is aware of the new branch.)

See if, when using the AnimateDiff Loader and when not using it, if you get the could not patch message. Thanks!

[sjuxax]

Seems to work here with SDXL beta. Definitely more detail with the new_injection branch, and fewer artifacts. One screenshot:

That's with LCM at 4 steps, plus a 4-step LCM hires fix. ^

Don't get the failed to patch message on new_injection, do get it on main.

[Kosinkadink]

Gotcha, thanks for testing it out. I'll do some tests locally to make sure there were no obvious regressions in VRAM usage, and if everything goes well, I'll merge the changes into main sometime in the next 24 hours.

[Pimax1]

thanks so much, I tried you new code, I think there might be a little improvement but on my side I still cant get lcm to work with animateDiff properly.

examples . this is LCM 4 steps, cfg 1.5, without animateDiff, proof that LCM can generate detailed stuff easily :

this is lcm 4 steps, cfg 1.5, with animateDiff. the background will always just be a blurry color mess and details are all gone. It doesnt matter how many seed I try.

now AnimateDiff without lcm (euler 20 steps, cfg 8):

prompt remained the same. I really cant get lcm + animateDiff to work. I am very sad.

[Kosinkadink]

Some things peeps have tried for LCM is to set the beta_schedule on the AD Loader to linear (HotshotXL/default), and to set the steps a little higher. And to avoid v2 based motion models. The v2 motion models contain a mid block, which seems to really not mesh well with LCM.

[Pimax1]

no luck for me sadly. with v1 i get same problem and shuterstock background on top of it. will try with sdxl quickly

[Kosinkadink]

Don't use sd_mm_v15 (not to be confused with the _v2 version), that one is the only model that almost always produces the shutterstock logo.

[Pimax1]

Thanks, I tried all this :

same thing always happens: simplistic background, all details gone. tried different prompts, removed my loras, controlnet and prompt travel to keep it simple. it doesnt matter, the background is always a problem.

just tried sdxl setup with animatediff sdxl + lcm sdxl. i must be doing something wrong because results are not even close to make any sense. but its the same scenario, in my case everything works great seperatly, the only combo i cant get to work is lcm + animatediff. :'( If I am the only one having this problem then, maybe I will try to reinstall from scratch.

[justanothernguyen]

I got the same problem, first run err, second run works, although the patched model seems broken, only low quality output. Probably because it erred while patching, so the code to eject didn't run.

[sjuxax]

On SDXL, you need to use the lcm-lora-sdxl LoRA and feed through a ModelSamplingDiscrete node set to LCM. I get acceptable results this way with LCM and SDXL. Haven't tried any SD1.x models.

[Kosinkadink]

After doing some testing, I can't merge the changes into main yet - there are issues with some stuff from VRAM not being cleaned up when changing between workflows, causing increased VRAM usage. For something like a 4090, not an issue, but would likely cause a lot of problems for anyone close to their VRAM cap. I will work on it this weekend to resolve that, and then I will merge into main.

[Two82]

Yes, as someone who was already running very close to cap on a 3060. After doing git pull and git checkout new_injection I started getting this error.

[AnimateDiffEvo] - INFO - Ejecting motion module hotshotxl_mm_v1.pth version None. ERROR:root:!!! Exception during processing !!! ERROR:root:Traceback (most recent call last): File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 131, in animatediff_sample orig_memory_required = model.model.memory_required # allows for "unlimited area hack" to prevent halving of conds/unconds ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1695, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'") AttributeError: 'SDXL' object has no attribute 'memory_required'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(*slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\SeargeSDXL\modules_legacy.py", line 93, in sample base_result = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1207, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample return original_sample(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 209, in animatediff_sample model.model.memory_required = orig_memory_required ^^^^^^^^^^^^^^^^^^^^ UnboundLocalError: cannot access local variable 'orig_memory_required' where it is not associated with a value

[Kosinkadink]

@Two82 in your case your ComfyUI is just outdated (by like a week). Updating to latest Comfy will get rid of that issue, that error is unrelated to memory usage.

[Two82]

ok thanks, I just assumed wrong due to unboundlocalerror 'orig_memory_required'. Thanks again!!!

[kacecarol]

Thank you. Should I shift to v14?

Some things peeps have tried for LCM is to set the beta_schedule on the AD Loader to linear (HotshotXL/default), and to set the steps a little higher. And to avoid v2 based motion models. The v2 motion models contain a mid block, which seems to really not mesh well with LCM.

[xerium9]

tldr: Changing the type of ControlNet (ComfyRoll -> Advanced-ControlNet) worked for me but I still get the errors in terminal.

Hey guys, I wanted to throw my 2 cents in. While I am not using a LCM (but I am going to try now that I know about it), I too am having the same errors while using AnimateDiff and ControlNet:

could not patch. key doesn't exist in model: diffusion_model.middle_block.2.in_layers.2.weight could not patch. key doesn't exist in model: diffusion_model.middle_block.2.emb_layers.1.weight could not patch. key doesn't exist in model: diffusion_model.middle_block.2.out_layers.3.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.2.1.conv.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.5.2.conv.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.8.2.conv.weight

However I also had this error:

Control type ControlNet may not support required features for sliding context window; use Control objects from Kosinkadink/Advanced-ControlNet nodes, or make sure Advanced-ControlNet is updated.

After making sure that cnet and everything was updated, I was still getting the same error.

For me, I found that I was using the ComfyRoll cnet stack going to the CR Apply Multi-Controlnet was the problem.

I essentially bypassed it and used the Advanced-Control Net nodes and that at least allowed the generation to take place:

I'm not sure if my cnet workaround is the right way to apply multiple controlnets but that's a different topic... I'd prefer to use the stack but I don't see one in the Advanced-Controlnet nodes

For now, I was able to generate successfully however the errors still come up. Here's the full log:

FETCH DATA from: D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager\extension-node-map.json got prompt [AnimateDiffEvo] - INFO - Loading motion module mm_sd_v15_v2.ckpt [AnimateDiffEvo] - INFO - Using fp16, converting motion module to fp16 Requested to load SD1ClipModel Loading 1 new model Prompt executed in 9.79 seconds got prompt [AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (48) greater than context_length 16. [AnimateDiffEvo] - INFO - Injecting motion module mm_sd_v15_v2.ckpt version v2. Requested to load BaseModel Requested to load ControlNet Requested to load ControlNet Loading 3 new models could not patch. key doesn't exist in model: diffusion_model.middle_block.2.in_layers.2.weight could not patch. key doesn't exist in model: diffusion_model.middle_block.2.emb_layers.1.weight could not patch. key doesn't exist in model: diffusion_model.middle_block.2.out_layers.3.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.2.1.conv.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.5.2.conv.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.8.2.conv.weight 100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [12:36<00:00, 37.83s/it] [AnimateDiffEvo] - INFO - Ejecting motion module mm_sd_v15_v2.ckpt version v2. [AnimateDiffEvo] - INFO - Cleaning motion module from unet. [AnimateDiffEvo] - INFO - Removing motion module mm_sd_v15_v2.ckpt from cache Prompt executed in 779.21 seconds

The generated images look decent but I will play around with more settings. To me, I don't see an issue other than the errors in the terminal.

I did not do any of the gitpull injection stuff mentioned earlier in the thread.

[sjuxax]

Any update on new_injection? I've been continuing to use that branch with a few cherry-picks from main. Would like to get back on mainline if possible.

[molkemon]

Thje new_injection branch does no longer work apparently as it generates the ModelSamplingConfig error it seems :/

[sjuxax]

It still works with these commits

git cherry-pick 810173d
git cherry-pick e93be26
git cherry-pick 36d72f3

[Fade07w]

Was a solution found with the could not patch error?

Hopping in to see as I encountered it today after fresh updates. Hadn't tried a LCM/AnimateDiff generation before today so no prior work for reference.

[Kosinkadink]

It still works, just could potentially work better (not by a lot though). There is no solution other than me refactoring the injection code to put in the motion module blocks in the right order (and removed) with respect to the unet patches. The endeavour requires careful monitoring of any potential VRAM increases, etc, so I want to put in a good amount of testing and work on it. I'm currently working on an Advanced-ControlNet overhaul, and maskable LoRA, so hopefully I can get a good refactor out into main in the next week or two.

[hben35096]

It seems that sd-webui-animatediff has solved this problem.

[kacecarol]

Thje new_injection branch does no longer work apparently as it generates the ModelSamplingConfig error it seems :/

I got exactly the same error. "ModelSamplingConfig" error or "could not patch. key doesn't exist in model" error, sigh.

[Kosinkadink]

Earlier today I started to officially work on a proper refactor of all motion model ejection, injection, and caching, so that it will use and respect built-in ComfyUI model patching conventions. I'll let ya guys know when I've got something ready to test (or, if everything looks fine to me, I might just merge into main and we'll test it live lol).

The result of the refactor will be 1) no more LCM patching issues (or any patching issues of any kind, hopefully), 2) proper device and dtype casting, so that the motion model is never in a state where it is not compatible with the dtype/device of the SD model, and 3) model will be able to be cached properly, meaning no loading motion model or motion loras from disk every time AD is used in a sampler.

This is going to be a fairly large undertaking, and if everything goes well, there should be no vram regressions - if there are, I'm sure I'll hear about them quickly upon release and we can sort them out as they come. This refactor, as big as it is, is going to save me a lot of headaches code wise too (I currently avoid doing any model caching after a sampling run to prevent issues on low vram devices that load models as metatensors), so Im happy to finally a chance to make it happen. I appreciate the patience thus far.

[AInseven]

@Kosinkadink It seems I encountered a different issue. LCM works well in a simple text2img workflow, but after adding animatediff, it encountered an error below. When I pressed the Queue Prompt a second time, KSampler started running, but the output had fewer details. And after that, the simple text2img also failed to produce proper output.

Error occurred when executing KSampler:

'VanillaTemporalModule' object has no attribute 'conv'

File "C:\ComfyUI-portable\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\nodes.py", line 1299, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\nodes.py", line 1269, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample return original_sample(*args, *kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 259, in animatediff_sample return wrap_function_to_inject_xformers_bug_info(orig_comfy_sample)(model, noise, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\model_utils.py", line 197, in wrapped_function return function_to_wrap(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\comfy\sample.py", line 93, in sample real_model, positive_copy, negative_copy, noise_mask, models = prepare_sampling(model, noise.shape, positive, negative, noise_mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\ComfyUI\comfy\sample.py", line 86, in prepare_sampling comfy.model_management.load_models_gpu([model] + models, model.memory_required([noise_shape[0] 2] + list(noise_shape[1:])) + inference_memory) File "C:\ComfyUI-portable\ComfyUI\comfy\model_management.py", line 383, in load_models_gpu unload_model_clones(loaded_model.model) File "C:\ComfyUI-portable\ComfyUI\comfy\model_management.py", line 328, in unload_model_clones current_loaded_models.pop(i).model_unload() File "C:\ComfyUI-portable\ComfyUI\comfy\model_management.py", line 311, in model_unload self.model.unpatch_model(self.model.offload_device) File "C:\ComfyUI-portable\ComfyUI\comfy\model_patcher.py", line 319, in unpatch_model comfy.utils.set_attr(self.model, k, self.backup[k]) File "C:\ComfyUI-portable\ComfyUI\comfy\utils.py", line 259, in set_attr obj = getattr(obj, name) ^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI-portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1695, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")

[YacratesWyh]

I've located it as the problem in v15_v2.ckpt by simple ablation test. Now we need further support on AnimateDiff. It's nothing about LCM.

[Kosinkadink]

My massive code rework/refactor is now out in main branch. Everything should look the same in terms of the nodes and majority of the generated animations. Notable features/side effects of my rework:

• Slightly lower VRAM usage (0.3-0.8GB) depending on workflow
• Motion model caching - speeds up consecutive sampling
• fp8 support (by casting in places that need to be casted)
• Model patches (like LCM) can be applied properly (no guarantees on improvements in generations though, might take some investigation to figure out why v2 models look weird with LCM)
• dtype and device mismatch edge cases should now be fixed
• New 'use existing' beta schedule added to allow any ModelSampling nodes before the AnimateDiff Loader to take effect - will use beta schedule as the ModelSampling patch overwise. Likewise, any ModelSampling nodes after the AnimateDiff Loader will overwrite the selected beta schedule, intuitively.

As mentioned in that list, LCM issue with v2 model i don't think is fixed yet. The LCM patch should work properly now, though, as the injection/ejection is in the right order now with the lora patching. I will look into v2 stuff in more detail during this week, now that my major reworks/feature updates to Advanced-ControlNet and AnimateDiff-Evolved are done now.

If you experience any new issues, let me know/open up new issues!

Note: it's out in main branch, NOT new_injection. I used the use_model_patcher branch to work on it before merging. Do not use new_injection branch for anything at this point, it's code is buggy and completely different from this rework that I did that's now in main branch.

[azoksky]

Awesome update! I think it willl be even more awesome if we fully integrate LCM lora features into it. I know u have been working on it and a few errors were fixed. I updated the local repo today. Yet, the images are choppy and lack any details. Just letting u know how far it progressed. I tested both temporaldiff-v1-animatediff.safetensors and mm-p_0.75.pth. They both suffer equally irrespective of versions.

[azoksky]

LCM lora/any distilled models produce inferior dark detailless images. Just a comparison.

[justanothernguyen]

LCM lora/any distilled models produce inferior dark detailless images. Just a comparison.

Quality tradeoff for speed.

Also with LCM you'll need to lower CFG, or rescale tonemap to avoid harsh contrast like this

[azoksky]

LCM lora/any distilled models produce inferior dark detailless images. Just a comparison.

Quality tradeoff for speed.

Also with LCM you'll need to lower CFG, or rescale tonemap to avoid harsh contrast like this

My dear, this is what I achieved with epirealism+LCM lora at 8 step lcm sampler. I didnt have to do any tone correction. The above image came out of the box. But, when combined with animatediff, it produces inferior dark images. Edit: once LCM is fully incorportated, it will be a game changer for sure. For now, without LCM lora..and yeah it takes awefully longer time to complete.