[Bug]: Recent commit causing ip-adapter_clip_sdxl_plus_vith to no longer work with ip-adapter-plus_sd15 [836b5c2e]

[silveroxides]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

After #2550 and/or #2551 the in title mentioned combination of preprocessor and model stopped working although they worked previously. I feel like the implementation of filtering and explicit errors based on the assumption that something doesn't/shouldn't work is rather questionable at best.

Steps to reproduce the problem

1. Fire up A1111 with version of CN after in title mentioned commits
2. select SD 1.5 checkpoint
3. Select preprocessor and model mentioned in title
4. get error
5. be sad

What should have happened?

1. Fire up A1111 with version of CN rolled back to before mentioned commits
2. Select SD 1.5 checkpoint
3. Select preprocessor and model mentioned in title
4. Get amazed to see it actually working
5. ???
6. PROFIT
7. Realize implementing code that deterministically filters or excludes any form of alternative use case without there being a bug that causes severe issues and without testing possible alternative use case scenarios is bad practice

Commit where the problem happens

webui: : v1.7.0-478-gb7f45e67 controlnet: 416c3450

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--skip-install --loglevel DEBUG --update-check --xformers --opt-sdp-attention --opt-split-attention --no-half-vae --upcast-sampling --disable-safe-unpickle --enable-insecure-extension-access --allow-code --listen --port 7860 --api --cors-allow-origins-regex="REDACTED" --deepdanbooru-projects-path "D:\stable-diffusion-webui\webui\models\deepdanbooru" --onnxtagger-path "D:\stable-diffusion-webui\webui\models\TaggerOnnx" --lyco-dir "D:\stable-diffusion-webui\webui\models\LyCORIS"

List of enabled extensions

Extensions

Console logs

part1

party2

Additional information

EDIT: Additional proof after I rolled back to the commit before

2024-02-14 19:36:18 INFO [modules.shared_state] Starting job task(dd71eum4kti2oob)

txt2img:
2024-02-14 19:36:18,599 - ControlNet - INFO - unit_separate = False, style_align = False
2024-02-14 19:36:18,954 - ControlNet - INFO - Loading model: ip-adapter-plus_sd15 [836b5c2e]
2024-02-14 19:36:18,960 - ControlNet - INFO - Loaded state_dict from [D:\stable-diffusion-webui\webui\extensions\sd-webui-controlnet\models\ip-adapter-plus_sd15.safetensors]
2024-02-14 19:36:19,195 - ControlNet - INFO - ControlNet model ip-adapter-plus_sd15 [836b5c2e] loaded.
2024-02-14 19:36:19,198 - ControlNet - INFO - Using preprocessor: ip-adapter_clip_sdxl_plus_vith
2024-02-14 19:36:19,198 - ControlNet - INFO - preprocessor resolution = 1024
2024-02-14 19:36:19,329 - ControlNet - INFO - ControlNet Hooked - Time = 0.7336032390594482
100%|██████████████████████████████████████████████████████████████████████████████████| 90/90 [00:21<00:00,  4.11it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 90/90 [00:23<00:00,  3.84it/s]
{"prompt": "", "all_prompts": [""], "negative_prompt": "", "all_negative_prompts": [""], "seed": 356465576, "all_seeds": [356465576], "subseed": 3257518762, "all_subseeds": [3257518762], "subseed_strength": 0, "width": 768, "height": 768, "sampler_name": "Euler a", "cfg_scale": 6, "steps": 90, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "Ultimate-NewEra-Freedom-alt2-fp32-no-ema-clip-fix", "sd_model_hash": "7339559dee", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Hypertile VAE": true, "Hypertile VAE max depth": 2, "Hypertile VAE max tile size": 64, "Hypertile VAE swap size": 24, "CFG Rescale phi": 0, "ControlNet 0": "Module: ip-adapter_clip_sdxl_plus_vith, Model: ip-adapter-plus_sd15 [836b5c2e], Weight: 1, Resize Mode: Just Resize, Low Vram: False, Processor Res: 1024, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", "Eta": 0.5}, "index_of_first_image": 0, "infotexts": ["Steps: 90, Sampler: Euler a, CFG scale: 6, Seed: 356465576, Size: 768x768, Model hash: 7339559dee, Model: Ultimate-NewEra-Freedom-alt2-fp32-no-ema-clip-fix, Hypertile VAE: True, Hypertile VAE max depth: 2, Hypertile VAE max tile size: 64, Hypertile VAE swap size: 24, CFG Rescale phi: 0, ControlNet 0: \"Module: ip-adapter_clip_sdxl_plus_vith, Model: ip-adapter-plus_sd15 [836b5c2e], Weight: 1, Resize Mode: Just Resize, Low Vram: False, Processor Res: 1024, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True\", Eta: 0.5, Version: v1.7.0-478-gb7f45e67"], "styles": [], "job_timestamp": "20240214193618", "clip_skip": 1, "is_using_inpainting_conditioning": false, "version": "v1.7.0-478-gb7f45e67"}
2024-02-14 19:36:43 INFO [modules.shared_state] Ending job task(dd71eum4kti2oob) (24.80 seconds)

Here are the input image as well as the resulting image.

[huchenlei]

We will add uniformed preprocessors like what is done in forge right now.

[2blackbar]

Some ipadapters do not work in forge... so yeah, that could break it even more

[silveroxides]

Some ipadapters do not work in forge... so yeah, that could break it even more

So some preprocessors are excluded from the main extension for AUTOMATIC1111/stable-diffusion-webui because they don't work in a fork of it? I was under the impression that the controlnet in forge was builtin and thought it would have code that differs compared to the regular extension. Also considering forge is a fork and as you say, some preprocessors do not work, then exclusion of them should not be handled on a main branch. Rather forge should have its own branch in this repo. Especially if any changes is made by AUTOMATIC1111 which would completely break for forge which forge would have to not merge. That should not be affecting the main branch.

[silveroxides]

We will add uniformed preprocessors like what is done in forge right now.

@huchenlei For real, can you stop self-assigning issues if you are not gonna give proper response and just abandon the issue leaving it open?

Also please give a proper reply to my last comment. It was the same thing with the lama preprocessor issue some months back which I posted including extensive demonstration of it and explaining why the cleaner extension is not the same thing.

[huchenlei]

@silveroxides Sorry about that. I was busy in the last month, and did not catch up on ControlNet maintenance. I have #2710 up to address your issue.