[ControlNet 1.1] The updating track.

[lllyasviel]

We will use this repo to track some discussions for updating to ControlNet 1.1.

[catboxanon]

(2) it seem that the previous "using only mid control for high-res" is broken of deleted?

I believe it was superseded by this PR: https://github.com/Mikubill/sd-webui-controlnet/pull/720

[lllyasviel]

about (1) when global average pooling is True, the controlnet should be only put on the cfg conditional side, otherwise the shuffle wont work very well This is without cfg-only Rightnow I have to so that it works but this will mess up all other controlnets so that we should have: when global average pooling is True, the controlnet should be only put on the cfg conditional side

[lllyasviel]

(2) it seem that the previous "using only mid control for high-res" is broken of deleted?

I believe it was superseded by this PR: #720

cool then rightnow what is the best practice for high res. fix?

[lllyasviel]

(6) oneflow needs dependency "pycocotools"

[lllyasviel]

after 5bad033 it still does not work

[lllyasviel]

cool thanks for the invite i will setup a development environment today

[catboxanon]

after 5bad033 it still does not work

It works for me, don't need to manually invert now. Usually after any extension update you'll need to hard restart the entire web UI to make sure all the modules are reloaded.

[lllyasviel]

after another commit it begin to work now

[lllyasviel]

right now the lineart works very well

[lllyasviel]

ok i think the 2 lineart model and 3 lineart preprocessors are perfect now.

[lllyasviel]

cool the 3 seg preprocessors are perfect

[lllyasviel]

ok two remaining problems (1) the visualization of softedge_pidinet is inverted (2) global average pooling should be linked to cfg conditional side ControlNet

[pkuliyi2015]

Can the Tile model work correctly now?

[lllyasviel]

no. Tile and Inpainting needs more time. Perhaps I can also work on inpaint

[catboxanon]

Point (2) was addressed in a comment in the PR, I can agree with you it should probably be enabled automatically: https://github.com/Mikubill/sd-webui-controlnet/pull/742#issuecomment-1509875513

shuffle requires Enable CFG-Based guidance to be enabled (in settings) for better result.

---

Can the Tile model work correctly now?

See above PR comment. Inpaint and Tile are not yet supported.

[lllyasviel]

openpose is perfect

congratulations

[continue-revolution]

no. Tile and Inpainting needs more time. Perhaps I can also work on inpaint

please let me know when inpaint is ready. my sam extension will connect to your inpaint. before that I'll try to connect to your semantic segmentation

[lllyasviel]

normalbae is perfect congratulations

[pkuliyi2015]

Yes I also want to the Tile model for upscaling. I got everything ready for draw / redraw images tile by tile, and the only thing need is a model trained on tiles.

Great jobs!

[lllyasviel]

hello where is the codes for preprocessor alias? I cannot find it

[Ratinod]

It would be great if a "force unload all controlNet preprocessors/models from VRAM" button exist.

[pkuliyi2015]

It would be great if a "force unload all controlNet preprocessors/models from VRAM" button exist.

I don't know, but I provide a Free GPU button in my extension. You may try use that button to see if it can unload controlnet stuffs?

[Ratinod]

I don't know, but I provide a Free GPU button in my extension. You may try use that button to see if it can unload controlnet stuffs?

It unload... but only little amount. (preprocessor) test method:

• set any image
• select preprocessor
• click "Preview annotator result"
• select other preprocessor
• click "Preview annotator result"
• continue until VRAM runs out. (Task Manager)

[lllyasviel]

update: guess mode rewritten, avg pool rewritten these are much more correct than before shuffle seed fixed now shuffle does not need more configs

[bropines]

update: guess mode rewritten, avg pool rewritten these are much more correct than before shuffle seed fixed now shuffle does not need more configs

Now you want me to change the branch to yours?

[lllyasviel]

shuffle + canny iron man, high-quality, extremely detailed Negative prompt: long body, low resolution, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 12345, Size: 512x768, Model hash: e1441589a6, Model: v1-5-pruned, ControlNet-0 Enabled: True, ControlNet-0 Module: none, ControlNet-0 Model: control_v11e_sd15_shuffle [526bfdae], ControlNet-0 Weight: 1, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 1, ControlNet-1 Enabled: True, ControlNet-1 Module: canny, ControlNet-1 Model: control_v11p_sd15_canny [d14c016b], ControlNet-1 Weight: 1, ControlNet-1 Guidance Start: 0, ControlNet-1 Guidance End: 1

canny

shuffle

result

[lllyasviel]

this should be the first really useable style transfer

[lllyasviel]

update: guess mode rewritten, avg pool rewritten these are much more correct than before shuffle seed fixed now shuffle does not need more configs

Now you want me to change the branch to yours?

you have a working branch?

[bropines]

update: guess mode rewritten, avg pool rewritten these are much more correct than before shuffle seed fixed now shuffle does not need more configs

Now you want me to change the branch to yours?

you have a working branch?

Now I'm on the ctrlnet 1.1 branch

[lllyasviel]

soem other attempts the style iron man, high-quality, extremely detailed Negative prompt: long body, low resolution, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality Steps: 50, Sampler: Euler a, CFG scale: 7, Seed: 12345, Size: 512x768, Model hash: e1441589a6, Model: v1-5-pruned, ControlNet-0 Enabled: True, ControlNet-0 Module: shuffle, ControlNet-0 Model: control_v11e_sd15_shuffle [526bfdae], ControlNet-0 Weight: 1, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 1, ControlNet-1 Enabled: True, ControlNet-1 Module: canny, ControlNet-1 Model: control_v11p_sd15_canny [d14c016b], ControlNet-1 Weight: 1, ControlNet-1 Guidance Start: 0, ControlNet-1 Guidance End: 1

[lllyasviel]

update: guess mode rewritten, avg pool rewritten these are much more correct than before shuffle seed fixed now shuffle does not need more configs

Now you want me to change the branch to yours?

you have a working branch?

Now I'm on the ctrlnet 1.1 branch

it is ok. u may take a look at my implementation and see if you want to fix any bug

[lllyasviel]

oh my god this style transfer is really killing

[lllyasviel]

@Mikubill I do not know how clip vision works. it seems already broken before my first commit. help needed

[pkuliyi2015]

Amazing work.

I’m also working for some new ideas in v2v and it seems to be good. Let’s change the game :)

[bropines]

I do not know how clip vision works. it seems already broken before my first commit. help needed

Can you make a manga model? I have a problem with the screenton, when inverted (or high sensitivity of the anotator) it causes inode artifacts

Sorry for the crooked table (I haven't figured out the markdown yet)

[lllyasviel]

I do not know how clip vision works. it seems already broken before my first commit. help needed

Can you make a manga model? I have a problem with the screenton, when inverted (or high sensitivity of the anotator) it causes inode artifacts

what is it?

[bropines]

I do not know how clip vision works. it seems already broken before my first commit. help needed

Can you make a manga model? I have a problem with the screenton, when inverted (or high sensitivity of the anotator) it causes inode artifacts

what is it?

On the 6th photo, the original is taken from the manga. At 4,5,7 the results of the anotator's work. The fact is that he is extremely bad at translating the manga screenton into the line version. 1,2,3 photos show the results of "coloring"

[lllyasviel]

update:

clip vision fixed

[0xprincess]

Hi everyone! I'm getting an unexpected keys error when trying to use fp16 v1.1 models. The full version works fine. How can I fix that?

[saint4626]

Error after the update, can anyone help?

[lllyasviel]

update and restart everything completely

[saint4626]

I updated and completely rebooted everything, it didn't work, annotator works, generation doesn't work

[lllyasviel]

can you see "cdcfa706"?

[saint4626]

yes

[lllyasviel]

then close the console and browser completely and the restart and it should work

[saint4626]

Finally, a strange mistake hmm, thank you!

[AbyszOne]

Ironically, the 3rd point is the first to be added, with this shuffle model, while no native batch was implemented whatsoever. Happy and sad face.jpg

[throttlekitty]

soem other attempts ... iron man, high-quality, extremely detailed Negative prompt: long body, low resolution, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality Steps: 50, Sampler: Euler a, CFG scale: 7, Seed: 12345, Size: 512x768, Model hash: e1441589a6, Model: v1-5-pruned, ControlNet-0 Enabled: True, ControlNet-0 Module: shuffle, ControlNet-0 Model: control_v11e_sd15_shuffle [526bfdae], ControlNet-0 Weight: 1, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 1, ControlNet-1 Enabled: True, ControlNet-1 Module: canny, ControlNet-1 Model: control_v11p_sd15_canny [d14c016b], ControlNet-1 Weight: 1, ControlNet-1 Guidance Start: 0, ControlNet-1 Guidance End: 1

Using your settings, I'm not getting similar images with Shuffle, the background is always a swirly mess. Or when using Shuffle by itself, I see that the output directly matches the flow that the annotator preview show.

[lllyasviel]

@throttlekitty make sure that your yaml file names and model file names are same

[throttlekitty]

That set me in the right direction, thanks. The extension places yaml files in its own models folder, but reads from webui\models\controlnet. I needed to have the yaml files in the same folder. It could also be that my webui\models\ is a symlink from another drive?