[Bug]: IP Adapter Weight Free Edit is not responsive

INTstinkt commented 7 months ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

If you manually type in the Weights for the ne Weight Type in IP Adapter XL to fast Controlnet doesn't get the changes and the Chart disappears. If this happens it actually dont register the Inputs and the Output picture is the same as if you didnt manually change the weight. It only happens for me if i type in the manual weights to fast. Little Workaround i found is deleting one diget, wait 1-2 sec then replacing it with a new digit till the chart is loaded back up, at this point it registered the input

Steps to reproduce the problem

Go to .... IP Adapter Weight Types Weights
Quickly Replace a digit with a new one
Chart disappeared and does not reappear
Generate a new picture which is the exact same picture it generated before changing the weights

What should have happened?

Registered the weight change and apply them to generation

Commit where the problem happens

webui: version: v1.9.0 • python: 3.10.11 • torch: 2.1.2+cu121 • xformers: 0.0.23.post1 • gradio: 3.41.2 • checkpoint: c9e3e68f89 controlnet: V1.1.445 IP Adapter Preprocessor: auto Model : Every Model

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--xformers

List of enabled extensions

Console logs

2024-04-21 07:36:15,279 - ControlNet - INFO - Loading model from cache: ip-adapter_sdxl [af81326a]
2024-04-21 07:36:15,280 - ControlNet - INFO - using mask
2024-04-21 07:36:15,290 - ControlNet - INFO - Using preprocessor: ip-adapter-auto
2024-04-21 07:36:15,290 - ControlNet - INFO - preprocessor resolution = 512
2024-04-21 07:36:15,311 - ControlNet - INFO - ControlNet Hooked - Time = 0.037128448486328125
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.65it/s]
postprocess_batch00%|██████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.64it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.58it/s]
2024-04-21 07:36:47,876 - ControlNet - INFO - unit_separate = False, style_align = False 20/20 [00:12<00:00,  1.64it/s]
2024-04-21 07:36:47,877 - ControlNet - INFO - Loading model from cache: ip-adapter_sdxl [af81326a]
2024-04-21 07:36:47,877 - ControlNet - INFO - using mask
2024-04-21 07:36:47,889 - ControlNet - INFO - Using preprocessor: ip-adapter-auto
2024-04-21 07:36:47,889 - ControlNet - INFO - preprocessor resolution = 512
2024-04-21 07:36:47,907 - ControlNet - INFO - ControlNet Hooked - Time = 0.03612494468688965
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.66it/s]
postprocess_batch00%|██████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.63it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.59it/s]
2024-04-21 07:37:19,744 - ControlNet - INFO - unit_separate = False, style_align = False 20/20 [00:12<00:00,  1.63it/s]
2024-04-21 07:37:19,744 - ControlNet - INFO - Loading model from cache: ip-adapter_sdxl [af81326a]
2024-04-21 07:37:19,746 - ControlNet - INFO - using mask
2024-04-21 07:37:19,758 - ControlNet - INFO - Using preprocessor: ip-adapter-auto
2024-04-21 07:37:19,758 - ControlNet - INFO - preprocessor resolution = 512
2024-04-21 07:37:19,787 - ControlNet - INFO - ControlNet Hooked - Time = 0.04917120933532715
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.64it/s]
postprocess_batch00%|██████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.64it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.57it/s]
2024-04-21 07:38:01,091 - ControlNet - INFO - unit_separate = False, style_align = False 20/20 [00:12<00:00,  1.64it/s]
2024-04-21 07:38:01,091 - ControlNet - INFO - Loading model from cache: ip-adapter_sdxl [af81326a]
2024-04-21 07:38:01,093 - ControlNet - INFO - using mask
2024-04-21 07:38:01,105 - ControlNet - INFO - Using preprocessor: ip-adapter-auto
2024-04-21 07:38:01,105 - ControlNet - INFO - preprocessor resolution = 512
2024-04-21 07:38:01,126 - ControlNet - INFO - ControlNet Hooked - Time = 0.040138959884643555
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.64it/s]
postprocess_batch00%|██████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.64it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.57it/s]
2024-04-21 07:38:27,876 - ControlNet - INFO - unit_separate = False, style_align = False 20/20 [00:12<00:00,  1.64it/s]
2024-04-21 07:38:27,876 - ControlNet - INFO - Loading model from cache: ip-adapter_sdxl [af81326a]
2024-04-21 07:38:27,876 - ControlNet - INFO - using mask
2024-04-21 07:38:27,887 - ControlNet - INFO - Using preprocessor: ip-adapter-auto
2024-04-21 07:38:27,887 - ControlNet - INFO - preprocessor resolution = 512
2024-04-21 07:38:27,908 - ControlNet - INFO - ControlNet Hooked - Time = 0.038132667541503906
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.65it/s]
postprocess_batch00%|██████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.62it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.58it/s]
2024-04-21 07:39:17,066 - ControlNet - INFO - unit_separate = False, style_align = False 20/20 [00:12<00:00,  1.62it/s]
2024-04-21 07:39:17,066 - ControlNet - INFO - Loading model from cache: ip-adapter_sdxl [af81326a]
2024-04-21 07:39:17,066 - ControlNet - INFO - using mask
2024-04-21 07:39:17,077 - ControlNet - INFO - Using preprocessor: ip-adapter-auto
2024-04-21 07:39:17,077 - ControlNet - INFO - preprocessor resolution = 512
2024-04-21 07:39:17,083 - ControlNet - INFO - IP-Adapter using advanced weighting [0.3, 0.2, 0.4, 0.8, 0.1, 0.3, 0.1, 0.4, 0.5, 0.7, 0]
2024-04-21 07:39:17,100 - ControlNet - INFO - ControlNet Hooked - Time = 0.04014015197753906
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.64it/s]
postprocess_batch00%|██████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.61it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.61it/s]
2024-04-21 07:40:08,128 - ControlNet - INFO - unit_separate = False, style_align = False 20/20 [00:12<00:00,  1.61it/s]
2024-04-21 07:40:08,129 - ControlNet - INFO - Loading model from cache: ip-adapter_sdxl [af81326a]
2024-04-21 07:40:08,129 - ControlNet - INFO - using mask
2024-04-21 07:40:08,139 - ControlNet - INFO - Using preprocessor: ip-adapter-auto
2024-04-21 07:40:08,139 - ControlNet - INFO - preprocessor resolution = 512
2024-04-21 07:40:08,159 - ControlNet - INFO - ControlNet Hooked - Time = 0.03612494468688965
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.61it/s]
postprocess_batch00%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.61it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.54it/s]
2024-04-21 07:40:54,931 - ControlNet - INFO - unit_separate = False, style_align = False 20/20 [00:12<00:00,  1.61it/s]
2024-04-21 07:40:54,931 - ControlNet - INFO - Loading model from cache: ip-adapter_sdxl [af81326a]
2024-04-21 07:40:54,931 - ControlNet - INFO - using mask
2024-04-21 07:40:54,942 - ControlNet - INFO - Using preprocessor: ip-adapter-auto
2024-04-21 07:40:54,942 - ControlNet - INFO - preprocessor resolution = 512
2024-04-21 07:40:54,947 - ControlNet - INFO - IP-Adapter using advanced weighting [0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]
2024-04-21 07:40:54,963 - ControlNet - INFO - ControlNet Hooked - Time = 0.037128448486328125
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.69it/s]
postprocess_batch00%|██████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.68it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.61it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.68it/s]

Additional information

Consol log kind of irrelevant for this one but the ones actually showing the weighting in the logs are the ones i waited between replacing the weights till the chart showed up again

huchenlei commented 7 months ago

A responsive free edit UI probably requires a pure front-end solution instead of gradio solution we have now.

bigmover commented 7 months ago

A responsive free edit UI probably requires a pure front-end solution instead of gradio solution we have now.

Hi guys! I found controlnet's preprocessor may take very long time(20s+) to preprocess。I can't figure out what cause it. Wound you mind to give some advice?

the input_image shape is (1152, 1152, 3) and threshold_a = -1/threshold_b = -1

huchenlei commented 7 months ago

A responsive free edit UI probably requires a pure front-end solution instead of gradio solution we have now.

Hi guys! I found controlnet's preprocessor may take very long time(20s+) to preprocess。I can't figure out what cause it. Wound you mind to give some advice?

the input_image shape is (1152, 1152, 3) and threshold_a = -1/threshold_b = -1

What preprocessor are you using?

bigmover commented 7 months ago

A responsive free edit UI probably requires a pure front-end solution instead of gradio solution we have now.

Hi guys! I found controlnet's preprocessor may take very long time(20s+) to preprocess。I can't figure out what cause it. Wound you mind to give some advice? the input_image shape is (1152, 1152, 3) and threshold_a = -1/threshold_b = -1

What preprocessor are you using?

It's "ip-adapter".

huchenlei commented 7 months ago

SDXL ipadapter uses clip-g as preprocessor which is slower than clip-h. You can try switch to ip-adapter_sdxl_vit-h which uses clip-h as preprocessor.

huchenlei commented 7 months ago

https://huggingface.co/h94/IP-Adapter/resolve/main/sdxl_models/ip-adapter_sdxl_vit-h.safetensors

bigmover commented 7 months ago

SDXL ipadapter uses clip-g as preprocessor which is slower than clip-h. You can try switch to ip-adapter_sdxl_vit-h which uses clip-h as preprocessor.

Besides I found clip would init everytime. why chang it （load/unload）? Can I keep it loaded in clipe_encoder(never unloaded)? whether can I keep it loaded in clip_encoder?

huchenlei commented 7 months ago

SDXL ipadapter uses clip-g as preprocessor which is slower than clip-h. You can try switch to ip-adapter_sdxl_vit-h which uses clip-h as preprocessor.

Besides I found clip would init everytime. why chang it （load/unload）? Can I keep it loaded in clipe_encoder(never unloaded)? whether can I keep it loaded in clip_encoder?

The behaviour changed in https://github.com/Mikubill/sd-webui-controlnet/pull/2754 when we ported forge's preprocessor structure, as forge aggressively unloads the preprocessor torch module after use (To save vram?).

If you think this behaviour is undesired, I can try restore previous behaviour.

bigmover commented 7 months ago

SDXL ipadapter uses clip-g as preprocessor which is slower than clip-h. You can try switch to ip-adapter_sdxl_vit-h which uses clip-h as preprocessor.

Besides I found clip would init everytime. why chang it （load/unload）? Can I keep it loaded in clipe_encoder(never unloaded)? whether can I keep it loaded in clip_encoder?

The behaviour changed in #2754 when we ported forge's preprocessor structure, as forge aggressively unloads the preprocessor torch module after use (To save vram?).

If you think this behaviour is undesired, I can try restore previous behaviour.

Thank you very much for your prompt and kind response！I want to keep the the instance of ClipVisionDetector in clip_encoder as I found it will cost extra time due to Initialize ClipVisionDetector with config(clip_g, clip_h, clip_vitl) everytime. But I'm not sure if this will cause any problems. Therefore, I would like to seek your valuable advice. Never unload

Mikubill / sd-webui-controlnet