Apply Image segmentation to mask a specific area for inpainting alternate to manual masking.

[Muhammad-Zubair]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

What if we Use model like yolov4 to detect objects in an image then create mask (or even delete the unwanted object or area) and then inpaint which will be much better and accurate. also image segmentation is much easier than text to mask or masking with brush, with just one click away let the model do object detection and masking so inpainting process will be like this:

-> detect an object with yolov4 or any other highly accurate model -> create mask with U-Net , FastFCN, MaskRCNN (which ever model full fill the purpose best) -> ( if it give us good result delete the object or area from image) -> inpaint the masked area ( with text to img)

Proposed workflow

1. Object Detection
2. Masking (or deletion of unwanted object or area)
3. Inpainting

[Ehplodor]

I'll add this to the list https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2940

[winterspringsummer]

I'm not sure if this is what you are looking for, but there is a custom scripts for selecting area by text - txt2mask https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts#txt2mask

[Muhammad-Zubair]

I'm not sure if this is what you are looking for, but there is a custom scripts for selecting area by text - txt2mask https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts#txt2mask image segmentation is much easier than text to mask with just one click away let the work done by image segmentation model or even yolov4 or yolov5 models like that can be used to: -> detect an object with yolov4 or any other highly accurate model -> create mask with U-Net , FastFCN, MaskRCNN (which ever model full fill the purpose best) -> inpaint the masked object