search

ototadana / sd-face-editor

Face Editor for Stable Diffusion
MIT License
1.03k stars 85 forks source link

XSeg, a better mask generator #143

Closed fgas453243 closed 1 year ago

fgas453243 commented 1 year ago

Hello. Thanks for the great extension release and the updates.

Is it possible to adopt XSeg as a better mask-generator than the one currently implemented?

XSeg is a mask generation mechanism mainly used in software called deepfacelab. XSeg identifies faces and the rest (bangs, sunglasses, whistles, microphones, etc.) and generates a mask that does not contain non-face obstructions. XSeg generates masks based on pre-trained models. You can train the model yourself or use the Generic-XSeg model shipped with deepfacelab. The model consists of the following 4 files. ・XSeg_256_opt.npy ・XSeg_256.npy ・XSeg_data.dat ・XSeg_default_options.npy

Here is an example using XSeg. Below is the source image. source

First, fix the source image using NoMaskGenerator. 00003-880432504 0-best quality, 4k, 8k, high quality, very high resolution, RAW photo,_1girl,_straight hair, long hair, black hair, dark purple ha

Then apply XSeg to the source image to generate the mask image. mask_xseg-dst

Applies an XSeg mask to the fixed image. xseg-applied_00003-880432504 0-best quality, 4k, 8k, high quality, very high resolution, RAW photo,_1girl,_straight hair, long hair, black hair, dark purple ha

Finally merge with the source image. merged_00003-880432504 0-best quality, 4k, 8k, high quality, very high resolution, RAW photo,_1girl,_straight hair, long hair, black hair, dark purple ha The face is fixed and the rest looks unfixed.

For comparison, here is an example by BiSeNetMaskGenerator. 00002-880432504 0-best quality, 4k, 8k, high quality, very high resolution, RAW photo,_1girl,_straight hair, long hair, black hair, dark purple ha tmpqbewoaag BiSeNetMaskGenerator can accurately identify faces, but it doesn't seem to take into account obstacles.

ototadana commented 1 year ago

Thanks for the interesting suggestions.

I will read the deepfacelab code to see how I can implement it.

fgas453243 commented 1 year ago

Thank you for your reply immediately. I'm looking forward to it.

ototadana commented 1 year ago

It has been a while, so I will update you on the progress.

XSeg proved difficult to incorporate as a library, so I looked for another approach. As a result of investigation, I found that it might be possible to combine the following:

Yesterday I was able to create a model using these. And I was able to create the following mask image using this model just now:

image

Finally, I am now ready to begin implementation. Please wait a little longer.

fgas453243 commented 1 year ago

Thank you for letting me know your progress along the way. I look forward to implementing it.

ototadana commented 1 year ago

The implementation is ready.

It is available by following these steps:

  1. Update to the latest version
  2. In the "Settings" tab of the Face Editor section, set the following settings:
    • Enable 'Search workflows in subdirectories'.
    • Enable 'openmmlab' option under "Additional components".
  3. Restart the Web UI on the terminal.
  4. select 'examples/openmmlab' as workflow.

Please try it.

fgas453243 commented 1 year ago

Thank you for implementing an excellent mask generator.

I tried mmseg mask generator. tmppe2bhhrv The face is fixed and the rest looks unfixed.

Thank you for always responding to user requests. I look forward to the future of sd-face-editor. Thank you.