Is this possible to be used with multiple open pose controlnet?`

[strouder]

I have controlnet 0 (with an openpose) + ControlNet 1 (another openpose)... I want to combine the two poses into 1 image divided in the center but the characters keep on merging into 1 mutation.

[Kanareika]

Quick dirty fix: put both your poses on one image and feed it into one openpose tab. Also, there is problem with Style Adapter (https://github.com/Mikubill/sd-webui-controlnet/discussions/700). Looks like the extensions don't mix well.

[Symbiomatrix]

@strouder I seem to be able to reproduce the phenomenon (much like #19, I'd say) with the BREAK syntax, yet with 2D + vertical everything seems to be working quite well. My guess is that controlnet might somehow be affected by the raw BREAK when it builds the mask (it seems to operate first, possibly due to directory names in the extensions call stack), whereas in 2D mode the conversion of the special keywords to BREAK only happens once regional's process is called later, thus controlnet is unaffected. But this needs closer inspection and I don't understand controlnet's workings or extension interactions nearly well enough, just a hunch. For now, 2D syntax is not a bad workaround, I think, if it works for you too.

@Kanareika Can you be more specific on the style issue? I don't use it so that's a bit vague. Does changing syntax as mentioned make any difference?

[Kanareika]

@Symbiomatrix I'm sorry for vague explanation. When Regional Prompter is active, style adapter changes does not appy at all:

1. Without RP active:
2. With RP active:

Changing syntax make no difference (same result with BREAK and ADD). "Base Ratio" also does not affect. But, if using "Latent" generation mode, first half of a batch is kinda affected.

[Symbiomatrix]

@Kanareika Are you getting this message when the run starts with region + style? "Warning: StyleAdapter and cfg/guess mode may not works due to non-batch-cond inference" Need to look into it further, but it's checking for a certain context shape in the hook.

Edit: In fact, controlnet has a "is_vanilla_samplers" flag for our problematic ddim & plms samples (#13), hum. And all it does with that is reverse the order of cond, uncond in the context chunk.

[Kanareika]

@Symbiomatrix No. For me, this message appears only if i don't batch generations (you can see on my screenshots, that i use batch of 2) with Style Adapter. Regional Prompter does not trigger this message.

[Symbiomatrix]

@Symbiomatrix No. For me, this message appears only if i don't batch generations (you can see on my screenshots, that i use batch of 2) with Style Adapter. Regional Prompter does not trigger this message.

Indeed, I have wondered about the batch; but that seems to be a sort of suppression, rather than a fix for the underlying issue; For me, when running style control with batchsize 1 and no region, the warning doesn't appear - is this the same for you?

[Kanareika]

is this the same for you?

Yes. I remember getting this message with bathsize 1, but it was probably due to exceeding the token limit of 75

[Symbiomatrix]

@Kanareika Alright, it's not (technically) a bug in regional but in controlnet - the BREAK keyword (to which all region separators are converted) is buggy in style mode, as is going over 75 tokens. You were there, so I guess you're better aware of the situation than I am. All I can tell is that hook.forward gets called twice instead of once, with x = 1x4x64x64 and context = 1x77x768 + 1x(77*clauses)x768, instead of x = 2x4x64x64 and context = 2x77x768 as normal, or x = 1x4x64x64 and context = 1x(77*clauses)x768 + 1x77x768 with other processors. I'm not really sure how breaks are handled, as I've only seen one reference to them in webui, and it doesn't turn up in debug. So I can only surmise it's an internal model thing. My naive guess, the extra cond thing should be appended to the second iteration instead of first due to the reversed order.

https://github.com/Mikubill/sd-webui-controlnet/issues/502

[Symbiomatrix]

@Kanareika Huh, whaddya know. It was indeed almost the exact same fix as regional's pos/neg split. I'm on an old version of controlnet, but I'll try to upgrade and see if it's still structured the same. However, it still doesn't appear to apply a style when regional is enabled. How they interact isn't really clear to me either. Maybe the addition to context is supposed to reach regional too, somehow.

Edit: Okay, we're in luck, the context gets passed to regional with the addendum. So I guess it's just a matter of taking it into consideration, maybe.

[Symbiomatrix]

Great success. This needs two PRs to work, one on hako's repo and the other on miku's. It's frankly quite a patchwork and there's probably a much better way to communicate the usage of style net, but it seems to get the job done in the basic instance.

[Symbiomatrix]

Here are the necessary PRs: https://github.com/Mikubill/sd-webui-controlnet/pull/717 (Done) https://github.com/hako-mikan/sd-webui-regional-prompter/pull/40