ControlLLLite issue with SDXL Animatediff

[finaluzi]

If you don't connect the model out, it works, but the output doesn't seem to use ControlLLLite.

If you connect to a model out, you get this error

I'm not sure if I'm connecting it incorrectly, but when using ControlLLLite, how exactly do I connect the in and out of the model? Is there an example? I am using sdxl

Thanks!

[Kosinkadink]

Can you send the full error log from the console, and also create the simplest workflow you can that has the error and send it here as well? It will help me debug it.

This is most likely a bug, the way you are connecting it when you get the error is the intended way to use it - ControlLLLite requires model patching which is why the model input and output are both needed to use it.

[finaluzi]

The full output:

got prompt model_type EPS adm 2816 Using pytorch attention in VAE Working with z of shape (1, 4, 32, 32) = 4096 dimensions. Using pytorch attention in VAE missing {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'} left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids']) Requested to load SDXLClipModel Loading 1 new model [Advanced-ControlNet] - INFO - ControlLLLite controlnet! [Advanced-ControlNet] - INFO - loaded J:\sd\ComfyUI_windows_portable\ComfyUI\models\controlnet\xl\kohya_controllllite_xl_blur.safetensors successfully, 136 modules [AnimateDiffEvo] - INFO - Loading motion module hsxl_temporal_layers.f16.safetensors via Gen2 [AnimateDiffEvo] - INFO - [Autoselect]: 'linear (HotshotXL/default)' beta_schedule for hsxl_temporal_layers.f16.safetensors:v1:HotshotXL:SDXL Requested to load AutoencoderKL Loading 1 new model [AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (16) greater than context_length 8. [AnimateDiffEvo] - INFO - Using motion module hsxl_temporal_layers.f16.safetensors:v1. Requested to load SDXL Requested to load AnimateDiffModel Loading 2 new models 0%| | 0/20 [00:02<?, ?it/s] ERROR:root:!!! Exception during processing !!! ERROR:root:Traceback (most recent call last): File "J:\sd\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "J:\sd\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "J:\sd\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "J:\sd\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1375, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) File "J:\sd\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1345, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "J:\sd\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 334, in motion_sample latents = wrap_function_to_inject_xformers_bug_info(orig_comfy_sample)(model, noise, *args, *kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\utils_model.py", line 216, in wrapped_function return function_to_wrap(args, kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\sample.py", line 100, in sample samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 712, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 618, in sample samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 557, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, self.extra_options) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 154, in sample_euler_ancestral denoised = model(x, sigmas[i] s_in, extra_args) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 281, in forward out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 271, in forward return self.apply_model(*args, kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 268, in apply_model out = sampling_function(self.inner_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed) File "J:\sd\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 370, in evolved_sampling_function cond_pred, uncond_pred = sliding_calc_cond_uncondbatch(model, cond, uncond, x, timestep, model_options) File "J:\sd\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 478, in sliding_calc_cond_uncond_batch sub_cond_out, sub_uncond_out = comfy.samplers.calc_cond_uncond_batch(model, sub_cond, sub_uncond, sub_x, sub_timestep, model_options) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 222, in calc_cond_uncond_batch output = model.apply_model(inputx, timestep, c).chunk(batch_chunks) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\model_base.py", line 85, in apply_model model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, *extra_conds).float() File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 847, in forward h = forward_timestep_embed(module, h, emb, context, transformer_options, time_context=time_context, num_video_frames=num_video_frames, image_only_indicator=image_only_indicator) File "J:\sd\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 98, in forward_timestep_embed x = layer(x, context, transformer_options) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\attention.py", line 613, in forward x = block(x, context=context[i], transformer_options=transformer_options) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\attention.py", line 440, in forward return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\util.py", line 189, in checkpoint return func(inputs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\attention.py", line 479, in _forward n, context_attn1, value_attn1 = p(n, context_attn1, value_attn1, extra_options) File "J:\sd\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_lllite.py", line 72, in call q = q + self.modules[module_pfx_to_q](q, self.control) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "J:\sd\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "J:\sd\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_lllite.py", line 215, in forward cx = torch.cat([cx, self.down(x)], dim=1 if self.is_conv2d else 2) RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 880 but got size 920 for tensor number 1 in the list. Prompt executed in 64.62 seconds

And a simple example:

ControlLLLite_test.json

Thanks!

[Kosinkadink]

Thanks! I have pushed a fix for the problem - if you pull latest Advanced-ControlNet updates, your error should go away.

After looking at another user's workflow as well, turns out the issue was an edge case within kohya's original controlllite comfy code that I based my code on. If size of input latents (which is image size // 8) were not divisible by 2 or 4, things would go wonky in the attention code causing the issue. I know explicitly look for these edge cases, and account for this loss of condhint height and/or width in specific controlllite layers by resizing the condhint when needed.

If the issue persists after update and comfy reboot, reopen this issue.