foivospar / Arc2Face

[ECCV 2024🔥] Arc2Face: A Foundation Model of Human Faces
https://arc2face.github.io/
MIT License
491 stars 32 forks source link

Can you confirm if other pretrained controlnets would be compatible with this model #13

Open trideeprath opened 2 months ago

trideeprath commented 2 months ago

I would assume that other pretrained controlnets would not be compatible with this model and would have to be finetuned. For example pose model would have to be finetuned.

foivospar commented 2 months ago

Given that we have fine-tuned both the unet and encoder, the pre-trained ControlNets designed for SD v1.5 are unlikely to work as plug-and-play and probably need fine-tuning (you could try them though, for instance lllyasviel/sd-controlnet-canny seems to work sometimes, but generally the quality is not very good). For pose, we have trained our own ControlNet on the FFHQ dataset conditioned on 3D pose images extracted by EMOCA.