lllyasviel / stable-diffusion-webui-forge

GNU Affero General Public License v3.0
5.31k stars 520 forks source link

[Feature Request]: Add support for PuLID #722

Open JoeyLearnsToCode opened 1 month ago

JoeyLearnsToCode commented 1 month ago

Is there an existing issue for this?

What would your feature do ?

see this discussion: https://github.com/Mikubill/sd-webui-controlnet/discussions/2841

PuLID is an ip-adapter alike method to restore facial identity. It uses both insightface embedding and CLIP embedding similar to what ip-adapter faceid plus model does. However, there is an extra process of masking out the face from background environment using facexlib before passing image to CLIP. PuLID also uses Eva CLIP instead of normal CLIP. In the attn overrides, PuLID also does something more than IPAdapter, as it zero pads the tensor and adds to ortho of hidden states. If you are interested you can read their paper on how this is handled.

Please add support for it.

Maybe ip-adapter-auto too.

Proposed workflow

merge it from original ContrctNet

Additional information

No response

greasebig commented 4 weeks ago

i advise: if your team are lazy to chase the advanced controlnet techenics, please give a one_click button to ban your built_in controlnet. currently, only removing your ban in json can't make controlnet extension works, it still has name conflict