[bug]: IP Adapter always results in black images.

[shanedk]

Is there an existing issue for this?

• [X] I have searched the existing issues

OS

Windows

GPU

cuda

VRAM

12GB

What version did you experience this issue on?

3.6.0rc4

What happened?

Every time I try using an IP Adapter, I get a black image as a result. It happens regardless of whether I use SD15 or SDXL models. It happens regardless of whether I use FP32 or FP16 (with the VAE fix on SDXL). It happens regardless of VAE. It persists even after redownloading the IP Adapter, or even using other IP Adapters. There are no errors on the console.

I've gone over every possibility with Kapa.ai on your Discord, and it exhausted every option except for posting a bug report here.

This issue has persisted since 3.5, but it's only now that I've been able to pin it down as a problem with the IP Adapter.

Screenshots

No response

Additional context

No response

Contact Details

shanedk on Discord

[lstein]

This seems to be an isolated problem on your system, as we've not received other complaints about IP Adapter and black images. Could you provide a few more details on your setup?

• What is your GPU make and model?
• What version of torch are you using?
• Are you using xformers?
• Have you modified the customized the attention settings or any of the more obscure image generation settings?
• Did IP adapters ever work for you?

I'm sure you've checked this, but do you have the ip_adapter.bin models installed? Your models directory should contain the following files:

./sd-1/ip_adapter/ip_adapter_plus_sd15
./sd-1/ip_adapter/ip_adapter_plus_sd15/ip_adapter.bin
./sd-1/ip_adapter/ip_adapter_sd15
./sd-1/ip_adapter/ip_adapter_sd15/ip_adapter.bin
./sd-1/ip_adapter/ip_adapter_plus_face_sd15
./sd-1/ip_adapter/ip_adapter_plus_face_sd15/ip_adapter.bin
./sdxl/ip_adapter/ip_adapter_sdxl
./sdxl/ip_adapter/ip_adapter_sdxl/ip_adapter.bin
./any/clip_vision/ip_adapter_sd_image_encoder
./any/clip_vision/ip_adapter_sdxl_image_encoder

[shanedk]

Thanks for replying! My answers:

1. RTX 3060 12GB VRAM
2. The one the script installed, which is 2.1.2+cu121.
3. The script installed xformers 0.0.23.post1, so I assume so.
4. No.
5. I only recently started working with them, so I can't say prior to 3.5. I can only say from that version to the current RC, it's not working for me.

And yes, all of those files are there.

[Millu]

Could you share one of the black images (with metadata) that is generated?

[shanedk]

I'm attaching two, one made with SD15 and the other with SDXL.

[Millu]

Thanks! I just tried generating an image by using all settings from your SD1.5 and this as the IP-Adapter image:

I was able to successfully generate an image. This was with both the latest RC (3.6.0rc5) and main

Are there any errors in the console when you generate an image with IP-Adapter?

[shanedk]

No, nothing in red or anything that appears out of the ordinary.

[shanedk]

I stand corrected: I tried using it with your image and got the black screen issue. But I did notice a line that said RuntimeWarning: invalid value encountered in cast images = (images * 255).round().astype("uint8") that I hadn't noticed before.

Here's everything it did:

[2024-01-10 09:54:30,944]::[InvokeAI]::INFO --> Loading model W:\AI\Itch\invokeai 3\invokeai3_standalone\invokeai\models\sd-1\main\stable-diffusion-v1-5, type sd-1:main:tokenizer
[2024-01-10 09:54:32,276]::[InvokeAI]::INFO --> Loading model W:\AI\Itch\invokeai 3\invokeai3_standalone\invokeai\models\sd-1\main\stable-diffusion-v1-5, type sd-1:main:text_encoder
[2024-01-10 09:54:56,291]::[InvokeAI]::INFO --> Loading model W:\AI\Itch\invokeai 3\invokeai3_standalone\invokeai\models\sd-1\main\stable-diffusion-v1-5, type sd-1:main:unet
[2024-01-10 09:55:41,910]::[InvokeAI]::INFO --> Loading model W:\AI\Itch\invokeai 3\invokeai3_standalone\invokeai\models\sd-1\main\stable-diffusion-v1-5, type sd-1:main:scheduler
[2024-01-10 09:55:43,409]::[InvokeAI]::INFO --> Loading model W:\AI\Itch\invokeai 3\invokeai3_standalone\invokeai\models\sd-1\ip_adapter\ip_adapter_sd15, type sd-1:ip_adapter
[2024-01-10 09:55:46,098]::[InvokeAI]::INFO --> Loading model W:\AI\Itch\invokeai 3\invokeai3_standalone\invokeai\models\any\clip_vision\ip_adapter_sd_image_encoder, type any:clip_vision
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:09<00:00,  2.08it/s]
[2024-01-10 09:57:22,227]::[InvokeAI]::INFO --> Loading model W:\AI\Itch\invokeai 3\invokeai3_standalone\invokeai\models\sd-1\main\stable-diffusion-v1-5, type sd-1:main:vae
W:\AI\Itch\invokeai 3\invokeai3_standalone\env\lib\site-packages\diffusers\image_processor.py:97: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")
[2024-01-10 09:57:50,521]::[InvokeAI]::INFO --> Graph stats: 14ecf5e2-ce62-4a33-9cbe-3f5957c3830b
[2024-01-10 09:57:50,522]::[InvokeAI]::INFO -->                           Node   Calls  Seconds  VRAM Used
[2024-01-10 09:57:50,525]::[InvokeAI]::INFO -->              main_model_loader     1     0.249s     0.000G
[2024-01-10 09:57:50,527]::[InvokeAI]::INFO -->                      clip_skip     1     0.115s     0.000G
[2024-01-10 09:57:50,527]::[InvokeAI]::INFO -->                         compel     2    23.106s     0.246G
[2024-01-10 09:57:50,528]::[InvokeAI]::INFO -->                          noise     1     1.026s     0.244G
[2024-01-10 09:57:50,529]::[InvokeAI]::INFO -->                  core_metadata     1     0.182s     0.244G
[2024-01-10 09:57:50,530]::[InvokeAI]::INFO -->                     ip_adapter     1     0.190s     0.244G
[2024-01-10 09:57:50,530]::[InvokeAI]::INFO -->                        collect     1     0.140s     0.244G
[2024-01-10 09:57:50,531]::[InvokeAI]::INFO -->                denoise_latents     1   145.303s     3.009G
[2024-01-10 09:57:50,531]::[InvokeAI]::INFO -->                            l2i     1    27.123s     2.844G
[2024-01-10 09:57:50,532]::[InvokeAI]::INFO -->               linear_ui_output     1     0.951s     0.303G
[2024-01-10 09:57:50,532]::[InvokeAI]::INFO --> TOTAL GRAPH EXECUTION TIME:  198.386s
[2024-01-10 09:57:50,535]::[InvokeAI]::INFO --> RAM used by InvokeAI process: 3.82G (+0.001G)
[2024-01-10 09:57:50,537]::[InvokeAI]::INFO --> RAM used to load models: 3.20G
[2024-01-10 09:57:50,538]::[InvokeAI]::INFO --> VRAM in use: 0.303G
[2024-01-10 09:57:50,538]::[InvokeAI]::INFO --> RAM cache statistics:
[2024-01-10 09:57:50,539]::[InvokeAI]::INFO -->    Model cache hits: 2
[2024-01-10 09:57:50,539]::[InvokeAI]::INFO -->    Model cache misses: 7
[2024-01-10 09:57:50,540]::[InvokeAI]::INFO -->    Models cached: 7
[2024-01-10 09:57:50,540]::[InvokeAI]::INFO -->    Models cleared from cache: 0
[2024-01-10 09:57:50,541]::[InvokeAI]::INFO -->    Cache high water mark: 3.20/20.00G
[2024-01-10 09:57:51,019]::[uvicorn.access]::INFO --> 127.0.0.1:50192 - "GET /api/v1/images/i/f27dd240-68b7-4ef8-9ccb-81b55d9f9041.png HTTP/1.1" 200
[2024-01-10 09:57:51,437]::[uvicorn.access]::INFO --> 127.0.0.1:50193 - "GET /api/v1/boards/?all=true HTTP/1.1" 200
[2024-01-10 09:57:51,467]::[uvicorn.access]::INFO --> 127.0.0.1:50192 - "GET /api/v1/images/i/f27dd240-68b7-4ef8-9ccb-81b55d9f9041.png/thumbnail HTTP/1.1" 200

[Millu]

What are the dimensions of the image you're using in your IP-Adapter?

[shanedk]

For the one that threw the error above, I was using the one you attached. For the original black images I attached, it's 512x512. I'm attaching that image in case you need more information about it.

[Millu]

I'm really at a loss here.. used your image and got this:

Things to help I can think of (sorry if the bot already gave you these):

• Some how the IP-Adapter model is messed up - can delete it and re-download from models.invoke.ai
• Re-install Invoke (you can just rerun the installer over your existing InvokeAI location)
• Something is wrong with the image (not sure what it could be)

This hasn't been reported before so I'm quite confused

[shanedk]

I've tried reinstalling the IP-Adapter model, and I've even tried installing additional IP-Adapter models. None of them work. I'll try reinstalling entirely and report back.

[shanedk]

Here are the steps I just took:

1. I blew away the IP-Adapter models on the disk.
2. I reran the installation.
3. I ran the model installer and put the IP-Adapter models back.
4. I ran InvokeAI and tried again.

No change. I still get the black image using the IP-Adapter along with the error image_processor.py:97: RuntimeWarning: invalid value encountered in cast images = (images * 255).round().astype("uint8").

Unless you have any other ideas, the only thing I can think to try at this point is a fresh install in a new location. (Hopefully it'll play nice with the existing one.)

[shanedk]

I had to wrestle with it a bit, but I finally got it to do a fresh install in a new directory. The IP Adapter is working fine now.

[Millu]

Glad you got it working!