Controlnet supported?

[jepjoo]

If I add Controlnet to "test ksampler refiner.json" workflow I get an error. Maybe I'm connecting it wrong?

!!! Exception during processing !!! Traceback (most recent call last): File "G:\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "G:\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "G:\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\nodes.py", line 49, in sample return (core.ksampler_with_refiner(model, positive, negative, refiner_model, refiner_positive, refiner_negative, latent_image, noise_seed, steps, refiner_switch_step, cfg, sampler_name, scheduler, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise), ) File "G:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\Fooocus\core.py", line 243, in ksampler_with_refiner samples = sampler.sample(noise, positive_copy, negative_copy, refiner_positive=refiner_positive_copy, File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\Fooocus\samplers_advanced.py", line 236, in sample samples = getattr(k_diffusionsampling, "sample{}".format(self.sampler))(self.model_k, noise, sigmas, File "G:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "G:\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 580, in sample_dpmpp_2m denoised = model(x, sigmas[i] * s_in, extra_args) File "G:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "G:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 323, in forward out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed) File "G:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "G:\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 125, in forward eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), kwargs) File "G:\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 151, in get_eps return self.inner_model.apply_model(*args, *kwargs) File "G:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 311, in apply_model out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed) File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\Fooocus\patch.py", line 296, in sampling_function_patched cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\Fooocus\patch.py", line 240, in calc_cond_uncond_batch c['control'] = control.get_control(inputx, timestep, c, len(cond_or_uncond)) File "G:\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 809, in get_control control = self.control_model(x=x_noisy, hint=self.cond_hint, timesteps=t, context=context, y=y) File "G:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "G:\ComfyUI_windows_portable\ComfyUI\comfy\cldm\cldm.py", line 279, in forward emb = self.time_embed(t_emb) File "G:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "G:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\container.py", line 217, in forward input = module(input) File "G:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, **kwargs) File "G:\ComfyUI_windows_portable\ComfyUI\comfy\ops.py", line 18, in forward return torch.nn.functional.linear(input, self.weight, self.bias) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

[hustille]

That looks like a sd controlnet model, not sdxl? The old models are not compatible, and there are very few sdxl ones. depth should be depth-zoe-xl-v1.0-controlnet.safetensors, afaik.

[jepjoo]

No it's definitely a SDXL model. I only have SDXL models on my ComfyUI and they all work on my other workflow with standard KSampler.

The filename I have changed myself.

[hustille]

How much video memory do you have? Can you run it with --force-gpu? On my system (8GB) running controlnet with a normal KSamplerAdvanced results in "out of memory". My guess is that with the combined sampler something gets squeezed out of GPU onto the CPU due to memory constraints which in the standard sampler forced to the GPU.

[jepjoo]

24GB (4090). There is no --force-gpu argument, maybe you meant --gpu-only? Tried with it and I get the same error.

Also tried with --disable-smart-memory earlier as Comfy just recently introduced a new smart memory management system and I thought it might be bugged but that didn't help either.

VRAM does fill up more with this KSampler than it does with the standard one so maybe there's something to it. When the crash happens VRAM usage is at 18.4GB/24GB when using a smaller controlnet model (controlnet-depth-sdxl-1.0-mid by Huggingface).

Would be interesting to know if Controlnet works for others or if the issue is on my end somehow.

[tastez]

ill check this out as my main use is always img2img and depth controlnet

[hustille]

Ok, --cpu works, if a bit slow. So it isn't a problem with the math, just the allocation of tensors to devices. I'll try to trace how Fooocus differs in this regard ...

[tastez]

also 4090 here and got the same exact error btw @jepjoo have you tried any approach to wire up both controlnet and also the SDXL prompt styler?

[hustille]

I did try some things too... The error is not dependent on SDXL, also happens with SD models, even with --gpu-only. On the other hand it did work once with SDXL, so apparently something gets randomly allocated and mostly lands on the CPU.

I compared the code with the standard comfy ksampler, but nothing stood out, especially nothing mentioning devices. Im stumped - tracing the problem deep into pytorch is beyond my expertise. Perhaps I'll play around with Fooocus a bit ...

[tastez]

i didnt realize your Kloader is not just for SDXL! i havent had issues with the regular KLoader and controlnet.

[jepjoo]

also 4090 here and got the same exact error btw @jepjoo have you tried any approach to wire up both controlnet and also the SDXL prompt styler?

Probably not but I'm not sure what prompt styler is.

[tastez]

also 4090 here and got the same exact error btw @jepjoo have you tried any approach to wire up both controlnet and also the SDXL prompt styler?

Probably not but I'm not sure what prompt styler is.

its an addon that gives you quick access to the SDXL prompt styles

[nistvan86]

Same error here. If I bypass the ControlNet conditioning the same graph works just fine. Also it works If I put another KSampler+Controlnet before the Foocus KSampler and continuing denoising that image without the ControlNet, but of course this limits it's usability.

[cerarslan]

Same here. This my error:

!!! Exception during processing !!! Traceback (most recent call last): File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\nodes.py", line 49, in sample return (core.ksampler_with_refiner(model, positive, negative, refiner_model, refiner_positive, refiner_negative, latent_image, noise_seed, steps, refiner_switch_step, cfg, sampler_name, scheduler, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise), ) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\Fooocus\core.py", line 243, in ksampler_with_refiner samples = sampler.sample(noise, positive_copy, negative_copy, refiner_positive=refiner_positive_copy, File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\Fooocus\samplers_advanced.py", line 236, in sample samples = getattr(k_diffusionsampling, "sample{}".format(self.sampler))(self.model_k, noise, sigmas, File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 137, in sample_euler denoised = model(x, sigma_hat * s_in, extra_args) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 323, in forward out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 125, in forward eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\external.py", line 151, in get_eps return self.inner_model.apply_model(*args, *kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 311, in apply_model out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\Fooocus\patch.py", line 296, in sampling_function_patched cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Fooocus_KSampler\sampler\Fooocus\patch.py", line 240, in calc_cond_uncond_batch c['control'] = control.get_control(inputx, timestep, c, len(cond_or_uncond)) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 809, in get_control control = self.control_model(x=x_noisy, hint=self.cond_hint, timesteps=t, context=context, y=y) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\cldm\cldm.py", line 279, in forward emb = self.time_embed(t_emb) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\container.py", line 217, in forward input = module(input) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, **kwargs) File "C:\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ops.py", line 18, in forward return torch.nn.functional.linear(input, self.weight, self.bias) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

I uploaded my workflow also. I tried all of three models (small,mid,large)

LastWF.txt