search

shiimizu / ComfyUI-TiledDiffusion

Tiled Diffusion, MultiDiffusion, Mixture of Diffusers, and optimized VAE
320 stars 23 forks source link

Bug when using with controlnet #25

Open aravindhv10 opened 6 months ago

aravindhv10 commented 6 months ago

Hi, when using tiled diffusion with more than one controlnet in series, I was running into errors. The workflow is here: https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/counter_example.json

The exact error is:

Requested to load ControlNet
Requested to load SDXL
Loading 3 new models
  0%|                                                                                                                         | 0/4 [00:07<?, ?it/s]
!!! Exception during processing !!!
Traceback (most recent call last):
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/nodes.py", line 1344, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/nodes.py", line 1314, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative
    return original_sample(*args, **kwargs)  # This code helps interpret error messages that occur within exceptions but does not have any impact on
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sam
    return orig_comfy_sample(model, *args, **kwargs)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/sample.py", line 37, in sample
    samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_d
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 755, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 657, in sample
    return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 644, in sample
    output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 623, in inner_sample
    samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 534, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
  File "/home/ubuntu/dbnew/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/k_diffusion/sampling.py", line 539, in sample_dpmpp_sde
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 272, in __call__
    out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 610, in __call__
    return self.predict_noise(*args, **kwargs)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 613, in predict_noise
    return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/comfy/samplers.py", line 258, in sampling_function
    out = calc_cond_batch(model, conds, x, timestep, model_options)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/custom_nodes/ComfyUI-TiledDiffusion/.patches.py", line 89, in calc_cond_batch
    output = model_options['model_function_wrapper'](model.apply_model, {"input": input_x, "timestep": timestep_, "c": c, "cond_or_uncond": cond_or_uncond}).chunk(batch_chunks)
  File "/home/ubuntu/dbnew/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/custom_nodes/ComfyUI-TiledDiffusion/tiled_diffusion.py", line 546, in __call__
    self.process_controlnet(x_tile.shape, x_tile.dtype, c_in, cond_or_uncond, bboxes, N, batch_id)
  File "/home/ubuntu/GITHUB/comfyanonymous/ComfyUI/custom_nodes/ComfyUI-TiledDiffusion/tiled_diffusion.py", line 342, in process_controlnet
    self.control_params[tuple_key][param_id][batch_id]=control.cond_hint
IndexError: list assignment index out of range

It seems that self.control_params (https://github.com/shiimizu/ComfyUI-TiledDiffusion/blob/main/tiled_diffusion.py#L292) in some circumstances is not initialized properly. I found a hacky way to fix this: https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314 wanted to know if this could be merged or there was a better way? Perhaps, these lines: https://github.com/shiimizu/ComfyUI-TiledDiffusion/blob/main/tiled_diffusion.py#L303-L306 can be rewritten.

Thanks!

gshawn3 commented 6 months ago

Thanks for reporting this! I ran into the same problem a week ago and hadn't had a chance to file an issue just yet. But I can also confirm that the node breaks when using more than one ControlNet in series.

aravindhv10 commented 6 months ago

I ran into these bugs when working with instant ID on huge images. Even after fixing this using the method I have mentioned, the issues remained on instant ID side. So once we have a proper solution here, I will report the issue in instant ID page!

Dervlex commented 6 months ago

Any idea to fix it ?

aravindhv10 commented 6 months ago

I managed to find a quick fix to the multiple controlnet problem in the links I had shared above (https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314). Donot know of a solution for instant ID yet.

If you just want to use tile diffusion with multiple controlnets for now, you can use the fork and branch from the link.

Dervlex commented 6 months ago

I managed to find a quick fix to the multiple controlnet problem in the links I had shared above (https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314). Donot know of a solution for instant ID yet.

If you just want to use tile diffusion with multiple controlnets for now, you can use the fork and branch from the link.

Duuuude Youre crazy! Thank you very much !

Can you text me on discord ? Would like to share my upscale workflow with you as thank you!

Dervlex is my name on Discord

Dervlex commented 6 months ago

I managed to find a quick fix to the multiple controlnet problem in the links I had shared above (https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314). Donot know of a solution for instant ID yet.

If you just want to use tile diffusion with multiple controlnets for now, you can use the fork and branch from the link.

But i get errors on 3 Control Nets now xD

2 works. 3 not :D

Dervlex commented 6 months ago

!!! Exception during processing!!! object of type 'NoneType' has no len() Traceback (most recent call last): File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\nodes.py", line 1344, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\nodes.py", line 1314, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample return original_sample(*args, *kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations. File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 47, in refcn_sample return orig_comfy_sample(model, args, kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\sample.py", line 37, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 755, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 657, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 644, in sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 623, in inner_sample samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 534, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, self.extra_options) File "C:\Users\dervl\ComfyUI Neu\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\k_diffusion\sampling.py", line 745, in sample_lcm denoised = model(x, sigmas[i] s_in, extra_args) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 272, in call out = self.inner_model(x, sigma, model_options=model_options, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 610, in call return self.predict_noise(*args, *kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 613, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 258, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion.patches.py", line 89, in calc_cond_batch output = model_options['model_function_wrapper'](model.apply_model, {"input": inputx, "timestep": timestep, "c": c, "cond_or_uncond": cond_or_uncond}).chunk(batch_chunks) File "C:\Users\dervl\ComfyUI Neu\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, **kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\tiled_diffusion.py", line 554, in call self.process_controlnet(x_tile.shape, x_tile.dtype, c_in, cond_or_uncond, bboxes, N, batch_id) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\tiled_diffusion.py", line 312, in process_controlnet while len(self.control_params[tuple_key][param_id]) <= batch_id: TypeError: object of type 'NoneType' has no len()

Vigilence commented 5 months ago

I managed to find a quick fix to the multiple controlnet problem in the links I had shared above (https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tiled_diffusion.py#L308-L314). Donot know of a solution for instant ID yet. If you just want to use tile diffusion with multiple controlnets for now, you can use the fork and branch from the link.

But i get errors on 3 Control Nets now xD

2 works. 3 not :D

Same. Just tested with 4. Doesn't work, but if I disable 2 of them then it will work. Nothing more then 2 will work.

aravindhv10 commented 5 months ago

Hi, sorry for the delayed reply. I made a few more changes in the same fork. It should now atleast work with 3 controlnets for SDXL (I tried with full canny, depth and TTPlanet tile). It was a minor bug in my code.

Vigilence commented 5 months ago

Hi, sorry for the delayed reply. I made a few more changes in the same fork. It should now atleast work with 3 controlnets for SDXL (I tried with full canny, depth and TTPlanet tile). It was a minor bug in my code.

I tried your new update with more then one controlnet and it wouldn't work. Disabling everything except one controlnet makes it work.

aravindhv10 commented 5 months ago

I was able to use it with 3 controlnets, can you check you are using the correct branch in my fork? You have can use "git switch fix_controlnet".

aravindhv10 commented 5 months ago

!!! Exception during processing!!! object of type 'NoneType' has no len() Traceback (most recent call last): File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\nodes.py", line 1344, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\nodes.py", line 1314, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample return original_sample(*args, *kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations. File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 47, in refcn_sample return orig_comfy_sample(model, args, kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\sample.py", line 37, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 755, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 657, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 644, in sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 623, in inner_sample samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 534, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, self.extra_options) File "C:\Users\dervl\ComfyUI Neu\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\k_diffusion\sampling.py", line 745, in sample_lcm denoised = model(x, sigmas[i] s_in, extra_args) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 272, in call out = self.inner_model(x, sigma, model_options=model_options, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 610, in call return self.predict_noise(*args, *kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 613, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\comfy\samplers.py", line 258, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion.patches.py", line 89, in calc_cond_batch output = model_options['model_function_wrapper'](model.apply_model, {"input": inputx, "timestep": timestep, "c": c, "cond_or_uncond": cond_or_uncond}).chunk(batch_chunks) File "C:\Users\dervl\ComfyUI Neu\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\tiled_diffusion.py", line 554, in call** self.process_controlnet(x_tile.shape, x_tile.dtype, c_in, cond_or_uncond, bboxes, N, batch_id) File "C:\Users\dervl\ComfyUI Neu\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\tiled_diffusion.py", line 312, in process_controlnet while len(self.control_params[tuple_key][param_id]) <= batch_id: TypeError: object of type 'NoneType' has no len()

Hi, what was the video memory you have ? Can you try reducing the tile diffusion batch size?

aravindhv10 commented 5 months ago

I managed to run this workflow:

https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/tile_diffusion_sr.json

on G5.2x large aws machine (24 gb gpu memory, ubuntu 22.04 native os python with venv) with the packages: https://github.com/aravindhv10/ComfyUI-TiledDiffusion/blob/fix_controlnet/versions

The xformers seem to help.

aravindhv10 commented 5 months ago

Hi, I also noticed that this error is very difficult to reproduce and depends on the resolution and other matters of the input image. I found that bypassing instant id resolves the error for me. Probably it comes from combination of insight face (face detection) and other complex interactions from instant id.

Vigilence commented 5 months ago

@aravindhv10 I am testing the update now. So far its working very great! I noticed some incompatibility with midas depth node, and zoe depth node, but it works with the regular depth node and leres node.

mikebilly commented 3 months ago

@aravindhv10 Hi, you might be doing something similar to what I want, but I keep not getting the desired output.

How to use Tiled-diffusion with controlnet depth and lineart and prompt to generate huge images?

I want to take in a huge image, and then apply controlnet depth, and use tiled diffusion to generate tile by tile to finally generate a high resolution image, same resolution as the original input image.

I want to achieve the results like these two examples below (using multidiffusion): image

edtjulien commented 2 months ago

Hey, should be related because now I have this error: File "D:\StableDiffusion\Packages\ComfyUI\comfy\ldm\modules\diffusionmodules\openaimodel.py", line 357, in apply_control if control is not None and name in control and len(control[name]) > 0: TypeError: argument of type 'ControlNet' is not iterable

The type of "control" should a tensor (or np array, I'm not sure), and when I print "control" before this error, it's a: <comfy.controlnet.ControlNet object at 0x740068723790>

I executed first a part of my workflow with 3 controlnet and tiled diffusion. And this error append after when I execute again my workflow but with only one controlnet.

Somewhere a conversion is not done? I can't find.

Edit: looks like someone had a similar issue: https://github.com/comfyanonymous/ComfyUI/issues/4809 I can see in log it uses Tiled Diffusion

aravindhv10 commented 2 months ago

Hi, sorry just saw the messages. Sorry @mikebilly I am not an expert at workflows, I work at a place where expert image editors take the task of coming up with workflows, I am only fixing issues.

@edtjulien Again, sorry we are not fully up to date with latest commit of tilediffusion. We use a specific commit which is known to work with atleast 4 controlnets (full controlnets, not control loras) and SDXL (including instant ids).

Yet to check this with flux either, we are on it though.

aravindhv10 commented 2 months ago

Can you share the workflow which leads to the error?

edtjulien commented 2 months ago

Hello ! Thanks for the fix and the follow up. The fix of @shiimizu solved my issue. So, no more workflow with a problem to share.

Thanks again for this!!