ControlNet and Kohya HRFix interop support

[RedDestiny]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

我经常使用高分辨率的图生图，会使用CN的lineArt和Tile两个模型的样子。而且我发现，小分辨率的时候（1000px），SD生成的图，光影和结构更加正确、优秀，超过2000px的时候就会变得很平、很容易出错的样子，无论是结构上，还是光影。

然后我使用了内置的Kohya HRFix，它的工作机制就是先缩放分辨率迭代出正确的结构，然后再提升会原本的分辨率，而且可以控制参与的步骤的样子。在文生图（1500x1500，甚至2000x2000以上），能得到结构正确且非常优秀的画面。 我就想着试着搭配CN模型一起，进一步控制画面的样子，然后就出现了报错的样子，在Kohya HRFix起作用的阶段中，CN就会出现 warning control could not be applied torch.Size([2, 1280, 11, 8]) torch.Size([2, 1280, 32, 22]) warning control could not be applied torch.Size([2, 1280, 11, 8]) torch.Size([2, 1280, 32, 22]) warning control could not be applied torch.Size([2, 1280, 11, 8]) torch.Size([2, 1280, 32, 22]) warning control could not be applied torch.Size([2, 1280, 22, 15]) torch.Size([2, 1280, 64, 43]) warning control could not be applied torch.Size([2, 1280, 22, 15]) torch.Size([2, 1280, 64, 43]) warning control could not be applied torch.Size([2, 640, 22, 15]) torch.Size([2, 640, 64, 43]) warning control could not be applied torch.Size([2, 640, 43, 29]) torch.Size([2, 640, 128, 86]) warning control could not be applied torch.Size([2, 640, 43, 29]) torch.Size([2, 640, 128, 86]) 然后并不起作用的样子。因为这个分辨率和CN设定的原本的分辨率不匹配的样子。

所以我特别希望CN也能自动匹配和Kohya HRFix的流程。

Proposed workflow

1. 到图生图/文生图，写好参数和设置
2. 启用Kohya HRFix Integrated，设置好参数
3. 启用CN，设置好参数和模型
4. CN能和HRFix一起正常运转

Additional information

No response

[Manchovies]

Also running into this. Seems to be Kohya HRFix that is messing this up. Happens on default fresh install as well.

2024-03-13 16:36:15,436 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE 2024-03-13 16:36:15,437 - ControlNet - INFO - Using preprocessor: canny 2024-03-13 16:36:15,437 - ControlNet - INFO - preprocessor resolution = 1024 2024-03-13 16:36:15,942 - ControlNet - INFO - Current ControlNet ControlNetPatcher: D:\Users\Spencer\Downloads\StabilityMatrix-win-x64\Data\Packages\Stable Diffusion WebUI Forge New\models\ControlNet\ControlNet\TencentARC-t2i-adapter-canny-sdxl.safetensors NeverOOM Enabled for VAE (always tiled) 2024-03-13 16:36:15,951 - ControlNet - INFO - ControlNet Method canny patched. To load target model SDXL Begin to load 1 model Reuse 1 loaded models [Memory Management] Current Free GPU Memory (MB) = 3321.5380859375 [Memory Management] Model Memory (MB) = 0.0 [Memory Management] Minimal Inference Memory (MB) = 1024.0 [Memory Management] Estimated Remaining GPU Memory (MB) = 2297.5380859375 Moving model(s) has taken 0.04 seconds 0%| | 0/30 [00:00<?, ?it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 3%|▎ | 1/30 [00:00<00:28, 1.03it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 7%|▋ | 2/30 [00:01<00:24, 1.13it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 10%|█ | 3/30 [00:02<00:22, 1.19it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 13%|█▎ | 4/30 [00:03<00:21, 1.23it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 17%|█▋ | 5/30 [00:04<00:20, 1.25it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 20%|██ | 6/30 [00:04<00:19, 1.26it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 23%|██▎ | 7/30 [00:05<00:18, 1.26it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 27%|██▋ | 8/30 [00:06<00:17, 1.27it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 30%|███ | 9/30 [00:07<00:16, 1.27it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) 33%|███▎ | 10/30 [00:08<00:15, 1.27it/s]warning control could not be applied torch.Size([2, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([2, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([2, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 320, 36, 63]) torch.Size([1, 320, 72, 126]) warning control could not be applied torch.Size([1, 640, 36, 63]) torch.Size([1, 640, 72, 126]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63]) warning control could not be applied torch.Size([1, 1280, 18, 32]) torch.Size([1, 1280, 36, 63])