[Feature]: inpaint preprocessor

[Flooosee]

Feature description

Can you implement the inpaint preprocessors ? They are quite usefull to do good fill (outpainting) while resizing. A bit lIke the photoshop outpaint. This is the only docs i've found, sorry if its not what you looking for :

• https://github.com/lllyasviel/ControlNet-v1-1-nightly?tab=readme-ov-file#controlnet-11-inpaint
• https://github.com/Mikubill/sd-webui-controlnet/discussions/1597 There was tree of them: inpaint_harmonious, inpaint_only, inpaint_only+lama.

The actual tool for outpainting is sadly not doing what this preprocessor is doing, just filling with coherent stuffs, without touching the current pic. Tell me if something is missing, it would be really appreciated if you manage to bring this in SDN :)

Version Platform Description

No response

[vladmandic]

as far as i understand, those are not really controlnet model preprocessors, they are just masking helpers - and sdnext already has those and more built-in (control module -> masking).

and not sure why you mention outpainting in this context? outpainting is different than inpainting.

[Flooosee]

Mh, that is a video where the preprocessor is used (link redirect on the good timecode) : https://youtu.be/wyDRHRuHbAU?t=410 For me he is doing outpainting, or maybe im wrong ? Is the tool you mentionned capable of doing this type of resizing ?

[vladmandic]

i've taken a quick look at video and i don't see any outpainting there. outpainting would be if i had 512x512 image and i want to make it 512x1024 by adding NEW content to left 256px and right of the original image. and yes, sdnext can do that as well.

what he's doing is simple img2img plus adding tile upscaling at the end of the video.

[Flooosee]

Not to offend but did we look the same video ? (Even his step 3 is named: "3. Outpaint using ControlNet", Haha). If we take a look a bit earlier you can see he generated a 768x768 pic, then in img2img he’s using the ControlNet inpaint_only+lama to Resize&Fill (so outpaint) to achieve a resolution of 1368x768. So yeah he’s adding new content to right and left. Also if you talking about the Outpaint feature in Image, yeah i’ve tested it, but it needs to have high denoising to get a proper image, and in return it’s changing the whole pic and not only the extra size, instead of this method who just accordingly of the og pic, adds new relative content on extras (maybe it’s possible to do the same with current tool, but i didn’t find out). So I may have misunderstood, but i don’t think so.

Here his workflow a bit sped up:

• generate image in 768x768

• sent it to img and fix problems with inpaint

• send it back to image, then change the resolution to 1368x768. Set denoising to 0.9. Check enable ControlNet, enable "Upload independent control image" checkbox and put the current image in the field. Set preprocessor to "inpaint_only+lama", and set model to "control_v11p_sd15_inpaint". Set resize mode to "Resize and Fill". Final step of step 3, generate till you have a good image.

• In image, upscale by 2 the previous final image generated using the ControlNet preprocessor "inpaint_global_harmonious" and still the same model.

• Final step. Set ControlNet preprocessor to "tile_resample" and model to "control_v11fle_sd15_tile". Now upscale the image using the extensions Ultimate SD Upscale.

Waiting for your reply, thx for your time.

[vladmandic]

I must admit I skipped over that. i much rather prefer when ppl explain what they want to achieve than point to go-there-I-want-that.

[Flooosee]

Sorry, but i literally explain the goal on first message... The achievement is to add these controlnets preprocessors to create nice outpainting following along the current pic without modiying it but add the extras. The video was only to give you a context of use, because you asked me why i talk about outpaint, wathever. My ask is only to improve SDN about outpainting, because right now i can't see a good uses of outpainting, as far as i know. Maybe we can already do this kind of outpainting, then i would be glad to know the method, but if not, is it possible to add these, or even to improve the current outpaint tool ? (also the preprocessors are good for inpaint fixes)

Current Workflow:

• Set a 768x768 pic, in Image tab.
• Set resize mode to outpaint, and set width to 1368.
• Set denoising to 0.9 otherwise it can't achieve a good result.
• Generate.

Result: The output is 1368x768, but changed the whole pic. Expected result: The output is 1368x768, the main pic isnt modified (768x768 part), and extras on right and left are coherent from the content of main pic.

[vladmandic]

sorry i wasn't clear - you did explain it in the previous writeup. keeping this open, will need some work.