Feature Request: txt2img ControlNet Mask for Inpaint Only + Lama

[duskydd]

https://github.com/Mikubill/sd-webui-controlnet/discussions/1597

Hi, I wondering if it is possible to add this ControlNet feature to the krita plugin since I don't see an option to add a mask in txt2img and img2img similarly to A1111. Not if this falls under "not supported" category since the feature can also used for outpainting.

[DrCyanide]

I think this is actually already supported, but I'll let you tell me if this works for you (I'm not getting great results in either the Web UI or in Krita, but I am getting results).

I don't think you need a mask for this ControlNet model to work, but rather a starting image. There's an option to select that image based on either your current selection, your active layer, or the whole canvas.

In my test, I had pasted in the image as a new layer, so I selected that layer in ControlNet. (I was re-using the image from my Web UI testing, you can achieve a similar effect by drawing a selection box before your first Txt2Img.) I set my ControlType to Inpaint and set all the other settings like I found in a tutorial on Inpaint only + Lama. (Someone reported a bug where the scrollbars don't show up in the dropdowns, so if you don't see Inpaint as an option and you don't see the scroll bar, try using your middle mouse button to scroll anyway).

Cyanic SD will set the Txt2Img generation size as whatever your canvas is or whatever your selected area is. So setting the canvas size or making a selection box acts the same as setting the Width and Height in Automatic 1111. I made a selection that matched the height of my layer, and clicked generate.

Here's the results I got. Not great, but obviously outpainted by ControlNet

[duskydd]

Thanks for the help! I was able to get the outpainting feature working fairly similarly to the webui but I wasn't able to figure how to use this controlnet to remove objects for the plugin.

[DrCyanide]

Ah, now I understand! I don't think I've ever needed to use the mask feature in ControlNet before, so I just ignored it.

I'll have to think about how to implement ControlNet masks in the UI, but it could be a quick fix.

[duskydd]

Thanks in advance!

[DrCyanide]

Minor update: It's going to be more annoying than I thought. I have the UI coded up, but ControlNet is more particular about the format for the mask than Stable Diffusion is. The masks I've been giving to Stable Diffusion aren't compatible with ControlNet, so I've got to figure out a way to convert them into a format that ControlNet will be happy with. I believe this is because the masks I'm generating are monochromatic (just black or white), while the masks that ControlNet wants are technically full color (even if they only use the black or white).

[DrCyanide]

Looks like I already had the conversion code laying around from some other ControlNet features. Added a checkbox above the image selection box to toggle to Mask mode, which works the same way as the Inpainting masks - The active layer will be treated as the mask, with the colored areas being selected, and the image below the mask will be sent as the non-masked image.

This is pushed, go ahead and give it a try.

[duskydd]

The UI is bit cluttered or overlaps when I enable "use mask" which isn't too much of a problem.

Thanks a lot for implementing it! It's working very well for me.

[DrCyanide]

Yeah... the overlap is a bug caused by having an animation for the collapsing categories (like Mask Settings, Inpaint Settings, Extensions, etc). I'll probably take out the animation at a later date to fix it.

[DrCyanide]

The UI is bit cluttered or overlaps when I enable "use mask" which isn't too much of a problem.

This UI overlapping bug should be fixed now.