[Feature]: allow FaceID together with T2I adapter

Issue Description

When using the Control endpoints in combination with the Face script, an error is produced. It happens both within controlnet and T2I Adapter.

Steps to reproduce:

Go to "Control" tab
Update an image for "Control Input"
Choose T2I and then the adapter OpenPose
Select the face script
Select a FaceID Model
Upload the image for FaceID
Click Generate

Screenshot 2024-06-18 at 07 38 22

This will result in the error which is included inside the Relevant log.

After this i was switching between controlnet and t2i adapter and got a different error after rerunning again. Not sure if this is pure because of toggling a different pipeline, but i include this error as well just in case:

05:39:50-716723 ERROR    Control pipeline failed: type=t2i adapter units=1 error=StableDiffusionAdapterPipeline.__call__() got an unexpected keyword argument 'shortcut'                  
05:39:50-717775 ERROR    Control: TypeError                                                                                                                                               
╭────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────╮
│ /home/ubuntu/ai_api/automatic_diffusers/modules/control/run.py:611 in control_run                                                                                                      │
│                                                                                                                                                                                        │
│   610 │   │   │   │   │   │                                                                                                                                                            │
│ ❱ 611 │   │   │   │   │   │   processed = p.scripts.run(p, *p.script_args)                                                                                                             │
│   612 │   │   │   │   │   │   if processed is None:                                                                                                                                    │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/scripts.py:483 in run                                                                                                                  │
│                                                                                                                                                                                        │
│   482 │   │   parsed = p.per_script_args.get(script.title(), args[script.args_from:script.args_to])                                                                                    │
│ ❱ 483 │   │   processed = script.run(p, *parsed)                                                                                                                                       │
│   484 │   │   s.record(script.title())                                                                                                                                                 │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/face/__init__.py:119 in run                                                                                                            │
│                                                                                                                                                                                        │
│   118 │   │   │   from modules.face.faceid import face_id                                                                                                                              │
│ ❱ 119 │   │   │   processed_images = face_id(p, app=app, source_images=input_images, model=ip_model, override=ip_override, cache=ip_cache, scale=ip_strength, structure=ip_structure)  │
│   120 │   │   │   processed = processing.Processed(p, images_list=processed_images, seed=p.seed, subseed=p.subseed, index_of_first_image=0) # manually created processed object        │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/face/faceid.py:228 in face_id                                                                                                          │
│                                                                                                                                                                                        │
│   227 │   │   │   │   debug(f"FaceID: {ip_model_dict}")                                                                                                                                │
│ ❱ 228 │   │   │   │   res = faceid_model.generate(**ip_model_dict)                                                                                                                     │
│   229 │   │   │   │   if isinstance(res, list):                                                                                                                                        │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/ip_adapter/ip_adapter_faceid.py:243 in generate                                                              │
│                                                                                                                                                                                        │
│   242 │   │                                                                                                                                                                            │
│ ❱ 243 │   │   images = self.pipe(                                                                                                                                                      │
│   244 │   │   │   prompt_embeds=prompt_embeds,                                                                                                                                         │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py:115 in decorate_context                                                           │
│                                                                                                                                                                                        │
│   114 │   │   with ctx_factory():                                                                                                                                                      │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                                                                                                         │
│   116                                                                                                                                                                                  │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: StableDiffusionAdapterPipeline.__call__() got an unexpected keyword argument 'shortcut'

Version Platform Description

05:28:27-799123 INFO     Starting SD.Next                                                                                                                                                 
05:28:27-801990 INFO     Logger: file="/home/ubuntu/ai_api/automatic_diffusers/sdnext.log" level=INFO size=1448809 mode=append                                                            
05:28:27-802963 INFO     Python version=3.10.12 platform=Linux bin="/home/ubuntu/ai_api/automatic_diffusers/venv/bin/python3" venv="/home/ubuntu/ai_api/automatic_diffusers/venv"         
05:28:27-813136 INFO     Version: app=sd.next updated=2024-06-17 hash=549f9ff7 branch=dev url=https://github.com/vladmandic/automatic/tree/dev ui=dev                                     
05:28:28-092087 INFO     Latest published version: a3ffd478e54c1735a1affc8b4760cef81594c293 2024-06-13T18:57:56Z                                                                          
05:28:28-094240 INFO     Platform: arch=x86_64 cpu=x86_64 system=Linux release=6.2.0-37-generic python=3.10.12                                                                            
05:28:28-095296 INFO     HF cache folder: /home/ubuntu/.cache/huggingface/hub

Relevant log output

05:31:12-511403 ERROR    Control pipeline failed: type=t2i adapter units=0 error=Input is in incorrect format. Currently, we only support <class 'PIL.Image.Image'>, <class               
                         'numpy.ndarray'>, <class 'torch.Tensor'>                                                                                                                         
05:31:12-512715 ERROR    Control: ValueError                                                                                                                                              
╭────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────╮
│ /home/ubuntu/ai_api/automatic_diffusers/modules/control/run.py:611 in control_run                                                                                                      │
│                                                                                                                                                                                        │
│   610 │   │   │   │   │   │                                                                                                                                                            │
│ ❱ 611 │   │   │   │   │   │   processed = p.scripts.run(p, *p.script_args)                                                                                                             │
│   612 │   │   │   │   │   │   if processed is None:                                                                                                                                    │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/scripts.py:483 in run                                                                                                                  │
│                                                                                                                                                                                        │
│   482 │   │   parsed = p.per_script_args.get(script.title(), args[script.args_from:script.args_to])                                                                                    │
│ ❱ 483 │   │   processed = script.run(p, *parsed)                                                                                                                                       │
│   484 │   │   s.record(script.title())                                                                                                                                                 │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/face/__init__.py:119 in run                                                                                                            │
│                                                                                                                                                                                        │
│   118 │   │   │   from modules.face.faceid import face_id                                                                                                                              │
│ ❱ 119 │   │   │   processed_images = face_id(p, app=app, source_images=input_images, model=ip_model, override=ip_override, cache=ip_cache, scale=ip_strength, structure=ip_structure)  │
│   120 │   │   │   processed = processing.Processed(p, images_list=processed_images, seed=p.seed, subseed=p.subseed, index_of_first_image=0) # manually created processed object        │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/face/faceid.py:228 in face_id                                                                                                          │
│                                                                                                                                                                                        │
│   227 │   │   │   │   debug(f"FaceID: {ip_model_dict}")                                                                                                                                │
│ ❱ 228 │   │   │   │   res = faceid_model.generate(**ip_model_dict)                                                                                                                     │
│   229 │   │   │   │   if isinstance(res, list):                                                                                                                                        │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/ip_adapter/ip_adapter_faceid.py:243 in generate                                                              │
│                                                                                                                                                                                        │
│   242 │   │                                                                                                                                                                            │
│ ❱ 243 │   │   images = self.pipe(                                                                                                                                                      │
│   244 │   │   │   prompt_embeds=prompt_embeds,                                                                                                                                         │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py:115 in decorate_context                                                           │
│                                                                                                                                                                                        │
│   114 │   │   with ctx_factory():                                                                                                                                                      │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                                                                                                         │
│   116                                                                                                                                                                                  │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py:1036 in __call__                   │
│                                                                                                                                                                                        │
│   1035 │   │   # 4. Preprocess image                                                                                                                                                   │
│ ❱ 1036 │   │   image = self.image_processor.preprocess(image)                                                                                                                          │
│   1037                                                                                                                                                                                 │
│                                                                                                                                                                                        │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/diffusers/image_processor.py:536 in preprocess                                                               │
│                                                                                                                                                                                        │
│    535 │   │   if not is_valid_image_imagelist(image):                                                                                                                                 │
│ ❱  536 │   │   │   raise ValueError(                                                                                                                                                   │
│    537 │   │   │   │   f"Input is in incorrect format. Currently, we only support {', '.join(str(x) for x in supported_formats)}"                                                      │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Input is in incorrect format. Currently, we only support <class 'PIL.Image.Image'>, <class 'numpy.ndarray'>, <class 'torch.Tensor'>

Backend

Diffusers

Branch

Dev

Model

SD 1.5

Acknowledgements

[X] I have read the above and searched for existing issues
[X] I confirm that this is classified correctly and its not an extension issue

vladmandic / automatic