[Feature Request] Batch process images using the ControlNet image input.

[FizzleDorf]

Hello. I was wondering if it was possible to run a batch process using the ControlNet input image rather than using batch img2img so the functionality could also be used in txt2img. I think there could be interesting results especially with animations. Bonus points if you get the ffmpeg commands in there as well so the user doesn't have to stitch everything together. From what I am seeing it can only read a single image for generating both txt2img and img2img.

[FizzleDorf]

I've looked through Issue #61. This could be a way to kill two birds with one stone.

[FizzleDorf]

I tried to use Video Loopback with ControlNet and since Video Loopback can't do batch img2img, only the first frame is affected. Hardly an implementation if you ask me. In the Loopback discussions I've asked to see if it was possible since there needs to be an input image in order for it to work. Would this still be considered if there isn't any way to get batch img2img for video loopback?

[marcsyp]

I was about to open an issue for this, and here are my comments. I believe this would be a HUGE feature.

---

I have been experimenting with using img2img with a source image and the same image to control all of my controlnet networks vs doing the same exact setup in txt2img (i.e., without the source image as a starting point) -- and I have found the txt2img to provide consistently better results, even after playing with weights, etc, than in img2img.

However, txt2img doesn't currently allow you to process images in batch (makes sense). Current workflow to process a large number of images is painful, involving replacing the source image for each control net one by one, clicking generate, and then moving to the next (at least this is less painful using the new tab interface, but still painful).

Proposed workflow Load a "ControlNet Batch" script Provide a directory of source images (ideally one directory for each controlnet, up to 10) For each image, the script replaces the controlnet source image (for each controlnet) with the corresponding image for that directory and runs a txt2img, moves to the next. Additional Features (Nice to haves) ability to vary the prompt by index position (directory for text files with different prompts with a way to assign each file to multiple index positions, with the A1111 input as a fallback?)

Another thought:

This may be possible to accomplish in img2img by simply providing a checkbox in the controlnet UI to ignore source image (may need to be a global checkbox?)

Just a few use cases to think about regarding this feature. These particularly related to animation, but could easily be applied to other use cases.

ANIMATION

Using the normal mapping with a background or foreground threshold to isolate subject matter. This works really well in txt2img, but fails miserably in img2img because the background content being isolated still provides weighted noise to the controlnet. using low resolution preprocessing to remove detail from a scene but maintain overall coherence, which works much better in txt2img. BATCH IMAGES

Using standard txt2img results with excellent composition as a low-fidelity proxy for a txt2img controlnet pass that adds high levels of detail without polluting the result with garbage pixel data. The ability to process a batch of content that has been preprocessed elsewhere (for instance, normal maps or depth maps) produced in external applications (Blender, etc), without needing to use the maps themselves as source data in img2img (which pollutes the controlnet result)

Thanks!

[ljleb]

This may be possible to accomplish in img2img by simply providing a checkbox in the controlnet UI to ignore source image (may need to be a global checkbox?)

Note that there is already a checkbox for this purpose in the extension settings (the "Skip img2img processing when using img2img initial image" checkbox). To use it, simply upload an image to the img2img image source while keeping the images of controlnet units empty.

[marcsyp]

This is great, thank you for that note! It was a hidden gem, I am now able to process large number of images using img2img and processing turned off -- success!

[jimys]

you save my day, thank you

[jimys]

This may be possible to accomplish in img2img by simply providing a checkbox in the controlnet UI to ignore source image (may need to be a global checkbox?)

Note that there is already a checkbox for this purpose in the extension settings (the "Skip img2img processing when using img2img initial image" checkbox). To use it, simply upload an image to the img2img image source while keeping the images of controlnet units empty.

after I use that option, the face still not as good as only txt2img+controlnet,

means if txt2img has a batch output will result better, I don't know why

[marcsyp]

Correct, I have had the same experience -- not just faces, but it seems that the source image still pollutes the i2i result in ways that the t2i doesn't. See my hack for accomplishing pure t2i batching here: https://github.com/lllyasviel/ControlNet/issues/171#issuecomment-1461294980

[4lt3r3go]

is there a way to drop an image in img2img and same image will be automatically used by all controlNET tabs? without the need to drop the image also in there everytime

[letsrock85]

stable-diffusion-webui\outputs\img2img-images\detected_maps\depth\2023-04-04