search

tencent-ailab / IP-Adapter

The image prompt adapter is designed to enable a pretrained text-to-image diffusion model to generate images with image prompt.
Apache License 2.0
5.27k stars 336 forks source link

Some questions about ipadapter faceid #190

Open Daming-TF opened 10 months ago

Daming-TF commented 10 months ago

Hello This is a great job！！ I want to ask about the specific details of this version of the training

1). Is the data set filtered from laion2B and coyo-700m like face plus?

2). Start training from scratch? Can 20W step have better results?

3). Is there any other processing, such as background noise, for the imported face reference picture?

whiterose199187 commented 10 months ago

Tagging along on this thread - is there code for training this available?

xiaohu2015 commented 10 months ago

1) Yes, we use the same training datasets with face plus 2) this one trained 50w steps 3) no, I just used insightface to get face embedding.

I will update a train script soon.

Daming-TF commented 10 months ago

For images with multiple face ids, should I input multiple id reference images to control the ROI region corresponding to bounding box through attention map during training？

xiaohu2015 commented 10 months ago

For images with multiple face ids, should I input multiple id reference images to control the ROI region corresponding to bounding box through attention map during training？

hi, the image in my datasest only contains a face ( use face detection model to filter). I also wonder that if it is helpful that use attention map mask.

xiaohu2015 commented 10 months ago

I add a training demo code.

Daming-TF commented 10 months ago

I add a training demo code.

I am replicating sdxl faceid. Is the training demo released so far incomplete? I see lora part is not included, right? I wonder if your code currently conflicts with the diffusers lora loading part of the logic, which uses the latest diffusers version (0.25.0.dev0). When reasoning, you must also manually set module.get_processor(return_deprecated_lora=True) in unet.atten_processors to module.get_processor(return_deprec ated_lora=False) ", in a nutshell my questions are three:

  1. Does the training demo lack lora?

  2. Is there a conflict in lora loading with diffuser currently inferred?

  3. lora loading problem Do you plan to fix it?

xiaohu2015 commented 10 months ago

hi, I include lora into https://github.com/tencent-ailab/IP-Adapter/blob/main/ip_adapter/attention_processor_faceid.py.

some disscutions at https://github.com/cubiq/ComfyUI_IPAdapter_plus/issues/145#issuecomment-1867138734