ControlNet used for this work?

[nlml]

Thanks for this great paper!

I noticed in Section 4.4 you say "we default to using RealisticVision 5.1 (RV5.1) and ControlNetMediaPipeFace".

Did you train this ControlNet yourself? Or is the checkpoint available online somewhere? I would like to use the same one for my research!

Thanks in advance!

[ZhenglinZhou]

Hi @nlml, thanks for your interest!

We use the checkpoint online, and you can find them in RV5.1 and ControlNetMediaPipeFace.

[nlml]

Thanks for the quick response!

I see. So just to confirm: you used the ControlNet adapter weights from ControlNetMediaPipeFace, but then used RV5.1 weights instead of SD1.5 weights as the model being 'controlled'? If so, I did not know you could do that.

[ZhenglinZhou]

Are you referring to the variances in the SD versions?

The RV5.1 is developed based on the SD1.5 (ref to the details of base model), and the ControlNetMediaPipeFace also includes a branch adapted to SD1.5. When ControlNet meets RV, it functions effectively (another example can be found in threestudio).

To use them, the following code may be helpful :

from diffusers import ControlNetModel, StableDiffusionControlNetPipeline
controlnet_name_or_path = "CrucibleAI/ControlNetMediaPipeFace"
pretrained_model_name_or_path = "stablediffusionapi/realistic-vision-51"

controlnet = ControlNetModel.from_pretrained(
                    controlnet_name_or_path,
                    subfolder="diffusion_sd15",
                    torch_dtype=self.weights_dtype,
                    cache_dir=self.cfg.cache_dir,
                )
pipe = StableDiffusionControlNetPipeline.from_pretrained(
            pretrained_model_name_or_path, controlnet=controlnet, **pipe_kwargs
        ).to(self.device)

I hope this answers your questions. If you have any other questions, feel free to leave a comment.

[nlml]

I see! Thanks a lot! That's very helpful.