[Feature] i find suitable scales for t2i adapter guess mode

[lllyasviel]

Because t2i adapter has fewer layers, weightings like 0.825**n will not work very well. After trying many possible weights, the below weights seem to work:

feature_64: 0.25 feature_32: 0.62 feature_16: 0.825 feature_8: 1.0

Let me know what you think or should i post some examples

[lllyasviel]

Because t2i adapter's encoder is lighter, their guess mode will look like this These results may be interesting in "pure guesses"; but with prompts, perhaps in some cases it will still improve the result a bit. But since we already have guess mode for some models, perhaps we should implement all.

[lllyasviel]

thanks very much! is it possible to put this in a bit higher priority? we are trying to perform some experiments in webui but if we can get this done we can simply reuse your codes. I tried this plugin today and this plugin is really awesome well done!

[Mikubill]

already added in https://github.com/Mikubill/sd-webui-controlnet/pull/262, will merge later after some tests

[lllyasviel]

thanks! looking forward!

[lllyasviel]

Hi the quality of results do not look similar to my results. seems something is not very correct.

[lllyasviel]

My results are achieved by disconnect t2i adapter in uncond, and then on the cond side we use 0.25, 0.62, 0.825, 1.0 as weights. 1.0 is for the 8*8 layer. I am also looking at the codes my self to detect differences.

[lllyasviel]

clearly not a problem of PLMS

[Mikubill]

Seems like adapter haven't disconnect on unconditional side, will fix later

[Mikubill]

misclicked. try with the latest commit

[lllyasviel]

let me try

[lllyasviel]

what happened?

[lllyasviel]

clearly a bug, but why is this even possible? ? ? i do not input prompts but image quality very high!

[lllyasviel]

find a way to reproduce it. load a img, generate, then click guesss mode, then generate, then change to ddim, then generate. boom! bug happens

[lllyasviel]

god. these steps not needed. just guess mode+ ddim+pidnet+t2i sketch

[lllyasviel]

god godess

[lllyasviel]

absolutely impossible

[lllyasviel]

I must know what happens.

[lllyasviel]

my controlnet also has this

amazing. what happens????

[Mikubill]

Tested with Euler a and it seems….fine? Maybe some assertions not fully compatible with non-k-diffusion samplers. (DDIM/PLMS)

[lllyasviel]

but what happens with ddim? this deserves a new paper! can we use this to extract the entire dataset of a diffusion model?

[Mikubill]

fixed in https://github.com/Mikubill/sd-webui-controlnet/commit/5a0fa4bd13bf49274e2075a3bd9909643104d8e9. Now it should generate similar result as other samplers does.

Root cause of that is vanilla samplers (DDIM/PMLS) using different input format than k-diffusion samplers, which is [uncond, cond] and [cond, uncond]. it's actually interesting: adding controlnet/adapter to uncond only could also produce good result. (and not related to input image)

before:

after:

[lllyasviel]

goddess. We need to delete this issue and not let others know.