Batch_size > 1 seems to break controlnet models

herval commented 1 year ago

I'm getting the weirdest bug with ControlNet 1.1 (used to work on an earlier release, broken for a week or so). This is through the sdapi/v1/txt2img endpoint.

I'm trying to use 2 controlnet steps (webui is configured for 2). Seed is fixed at 12345.

I'm using the following configuration (excerpt from the logs):

Loading preprocessor: scribble_xdog
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 512
raw_W = 512
target_H = 512
target_W = 512
estimation = 512.0
preprocessor resolution = 512
Loading model from cache: t2iadapter_style_sd14v1 [202e85cc]
Loading preprocessor: clip_vision
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 512
raw_W = 512
target_H = 512
target_W = 512
estimation = 512.0
preprocessor resolution = 512
Data shape for DDIM sampling is (1, 4, 64, 64), eta 0.0

Things work okay-ish when I set the batch_size to 1. Here's an example:

If I keep the exact same parameters but change the batch_size to 4, images come out more and more distorted on each iteration. Here's the outputs in that case:

It's as if the batch_size is somehow re-feeding the outputs on the batch?

herval commented 1 year ago

Actually, I noticed an even weirder issue with the shuffle model:

if I call the API with only the shuffle model (in my case, passing the second controlnet as disabled), the generation only works if the batch_size is 1:

Loading model from cache: control_v11e_sd15_shuffle_fp16
Loading preprocessor: shuffle
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 512
raw_W = 512
target_H = 512
target_W = 512
estimation = 512.0
preprocessor resolution = 512
Data shape for DDIM sampling is (1, 4, 64, 64), eta 0.0

Setting batch_size to anything else gets me a completely random image.

So I suspect the issue isn't with an interaction between cn + t2i models (my original suspicion), but rather something in the way the batch_size is being handled?

herval commented 1 year ago

For reference, the same issue happens through the webui: