[Bug]: Produces Black Images Only

DavidDragonsage commented 3 months ago

Checklist

[X] The issue has not been resolved by following the troubleshooting guide
[X] The issue exists on a clean installation of Fooocus
[X] The issue exists in the current version of Fooocus
[X] The issue has not been reported before recently
[ ] The issue has been reported before but has not been fixed yet

What happened?

SimpleSDXL is only producing purely black images.

I get this error just after generation begins:

0/30 [00:00<?, ?it/s]C:\SimpleAI\SimpleSDXL\modules\core.py:257: RuntimeWarning: invalid value encountered in cast x_sample = x_sample.cpu().numpy().clip(0, 255).astype(np.uint8)

Steps to reproduce the problem

1 Enter a simple prompt, such as "the cat in the hat" 2 Press Generate 3 The preview image is black and so is the final image, otherwise generation progress seems normal

What should have happened?

Fooocus should have produced an image of a "cat in a hat".

What browsers do you use to access Fooocus?

Mozilla Firefox

Where are you running Fooocus?

Locally

What operating system are you using?

Windows 10

Console logs

SimpleSDXL: Already up-to-date, 923f1a9
[System ARGV] ['SimpleSDXL\\entry_with_update.py', '--models-root', 'C:/SimpleModels', '--output-path', 'E:/stable-diffusion-webui/outputs']
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Fooocus version: 2.5.5
Comfy version: 233c67a
SimpleSDXL version: v20240820.923f1a9
Checking ...
[SimpleAI] GPU: NVIDIA GeForce GTX 1650, RAM: 28589MB, SWAP: 37723MB, VRAM: 4096MB, DiskFree: 30861MB, CUDA: 12.5
[SimpleAI] local_did/本地身份ID: 3mmoCdS86ta9c9yFptrabUTF2WLb
Load config data from C:\SimpleAI\SimpleSDXL\config.txt.
Loaded preset: C:\SimpleAI\SimpleSDXL\presets\default.json
The path_models_root: C:\SimpleModels
The path_output: E:\stable-diffusion-webui\outputs
Total VRAM 4096 MB, total RAM 28589 MB
Trying to enable lowvram mode because your GPU seems to have 4GB or less. If you don't want this use: --always-normal-vram
xformers version: 0.0.27
Set vram state to: LOW_VRAM
Always offload VRAM
Device: cuda:0 NVIDIA GeForce GTX 1650 : native
VAE dtype: torch.float32
Failed to load config key: {"default_engine": {}} is invalid; will use {"default_engine": {}} instead.
[SimpleAI] The path of models_info file: C:\SimpleModels\models_info.json
[Cleanup] Attempting to delete content of temp dir C:\Users\buyer\AppData\Local\Temp\fooocus
[Cleanup] Cleanup successful
Using xformers cross attention
Refiner unloaded.
model_type EPS
UNet ADM Dimension 2816
[Wildcards] Refresh and Load 27/27 wildcards: animal, bodytypes, buildings, chairs, color, containers, coverings, flora, flower, focus, foods, hairstyles, jobs, materials, minilocations, moods, nationality, objects, objectstohold, outfits, patterns, poses, rpg, shotsizes, space, timeperiods, vehicles.
[OneButtonPrompt] The customized module:OneButtonPrompt is initializing ...
IMPORTANT: You are using gradio version 3.41.2, however version 4.29.0 is available, please upgrade.
--------
[Comfyd] WARNING: cuda malloc enabled but not supported.
[Comfyd] Fix and patch for torch in windows
Running on local URL:  http://192.168.1.69:8186

To create a public link, set `share=True` in `launch()`.
[Comfyd] Device: cuda:0 NVIDIA GeForce GTX 1650 : cudaMallocAsync
[Gallery] Refresh_output_catalog: loaded 322 images_catalogs.
[Comfyd] [Prompt Server] web root: C:\SimpleAI\SimpleSDXL\comfy\web
   0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-Easy-Use
Comfy-Easy-Use v1.2.1: Loaded
[Comfyd] Device: cuda:0 NVIDIA GeForce GTX 1650 : cudaMallocAsync
[Comfyd] A new version of Albumentations is available: 1.4.14 (you have 1.4.7). Upgrade using: pip install --upgrade albumentations
Please 'pip install apex'
[Comfyd] Import times for custom nodes:
[Comfyd]    0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\websocket_image_save.py
[Comfyd]    0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\AIGODLIKE-COMFYUI-TRANSLATION
[Comfyd]    0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-FluxExt-MZ
[Comfyd]    0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI_essentials
[Comfyd]    0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI_IPAdapter_plus
[Comfyd]    0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-layerdiffuse
[Comfyd]    0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI_UltimateSDUpscale
[Comfyd]    0.0 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-IC-Light
[Comfyd]    0.1 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-Kolors-MZ
[Comfyd]    0.1 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-KwaiKolorsWrapper
[Comfyd]    0.1 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI_bitsandbytes_NF4
[Comfyd]    0.1 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI_ExtraModels
[Comfyd]    0.1 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-Advanced-ControlNet
[Comfyd]    0.1 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\PuLID_ComfyUI
[Comfyd]    0.4 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-KJNodes
[Comfyd]    0.4 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-BrushNet
[Comfyd]    0.9 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-Easy-Use
[Comfyd]    4.1 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI_InstantID
[Comfyd]    4.3 seconds: C:\SimpleAI\SimpleSDXL\comfy\custom_nodes\ComfyUI-SUPIR
[Comfyd]
[Comfyd]
WARNING: this card most likely does not support cuda-malloc, if you get "CUDA error" please run ComfyUI with: --disable-cuda-malloc

[Comfyd] Starting Comfyd server!

Using xformers attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using xformers attention in VAE
[Topbar] Reset_context: preset=default-->realistic-new, theme=dark, lang=default
Loaded preset: C:\SimpleAI\SimpleSDXL\presets\realistic-new.json
[Comfyd] Comfyd freeing!
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
Base model loaded: E:\stable-diffusion-webui\models\Stable-diffusion\juggernaut-X-RunDiffusion-NSFW.safetensors
VAE loaded: None
Request to load LoRAs [('sd_xl_offset_example-lora_1.0.safetensors', 0.1)] for model [E:\stable-diffusion-webui\models\Stable-diffusion\juggernaut-X-RunDiffusion-NSFW.safetensors].
Loaded LoRA [E:\stable-diffusion-webui\models\Lora\sd_xl_offset_example-lora_1.0.safetensors] for UNet [E:\stable-diffusion-webui\models\Stable-diffusion\juggernaut-X-RunDiffusion-NSFW.safetensors] with 788 keys at weight 0.1.
Fooocus V2 Expansion: Vocab with 642 words.
Fooocus Expansion engine loaded for cpu, use_fp16 = False.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
Started worker with PID 9208
App started successful. Use the app with http://192.168.1.69:8186/ or 192.168.1.69:8186
[Fooocus] GPU memory: max_reserved=0.002GB, max_allocated=0.000GB, reserved=0.000GB, free=3.221GB, free_torch=0.000GB, free_total=3.221GB, gpu_total=4.000GB, torch_total=0.000GB
[Fooocus] GPU memory: max_reserved=0.000GB, max_allocated=0.000GB, reserved=0.000GB, free=3.221GB, free_torch=0.000GB, free_total=3.221GB, gpu_total=4.000GB, torch_total=0.000GB
[TaskEngine] Task_class:Fooocus, Task_name:realistic-new, Task_method:text2image
[TaskEngine] Enable Fooocus backend.
[Comfyd] Comfyd freeing!
[Parameters] Adaptive CFG = 7
[Parameters] CLIP Skip = 2
[Parameters] Sharpness = 8
[Parameters] ControlNet Softness = 0.25
[Parameters] ADM Scale = 1.5 : 0.8 : 0.3
[Parameters] Seed = 8061974000370093222
[Parameters] CFG = 7
[Fooocus] Loading control models ...
[Parameters] Sampler = dpmpp_2m - karras
[Parameters] Steps = 30 - 18
[Fooocus] Initializing ...
[Fooocus] Loading models ...
Refiner unloaded.
model_type EPS
UNet ADM Dimension 2816
Using xformers attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using xformers attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
Base model loaded: C:\SimpleModels\checkpoints\RealvisXLv40_v40Bakedvae.safetensors
VAE loaded: None
Request to load LoRAs [('sd_xl_offset_example-lora_1.0.safetensors', 0.5)] for model [C:\SimpleModels\checkpoints\RealvisXLv40_v40Bakedvae.safetensors].
Loaded LoRA [E:\stable-diffusion-webui\models\Lora\sd_xl_offset_example-lora_1.0.safetensors] for UNet [C:\SimpleModels\checkpoints\RealvisXLv40_v40Bakedvae.safetensors] with 788 keys at weight 0.5.
Requested to load SDXLClipModel
Loading 1 new model
[Fooocus] Processing prompts ...
[Wildcards] Copmile text in prompt to arrays: the cat in the hat -> arrays:[], mult:0
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] the cat in the hat, bright background, cute, professional quality, dramatic light, dynamic composition, attractive, lively, elegant, intricate, detailed, very impressive, highly detail, creative, color spread, extremely fine, cinematic, enhanced, great still, expressive, beautiful, symmetry, perfect, artistic, wonderful, full coherent, sharp, pretty, inspired, vibrant colors
[Fooocus] Preparing Fooocus text #2 ...
[Prompt Expansion] the cat in the hat, bright, still, cinematic, dynamic, dramatic ambient, detailed, intricate, elegant, highly contrasted, sharp focus, professional fine detail, perfect composition, colorful, epic, best, modern, futuristic, contemporary, creative, color rich, attractive, cute, pretty, pure, cool, trendy, calm, awesome, amazing, background, crisp
[Fooocus] Preparing Fooocus text #3 ...
[Prompt Expansion] the cat in the hat, background still, cinematic, epic composition, dramatic light, detailed, warm colors, calm, magic, scenic, deep, rich vivid color, intricate, elegant, highly complex, coherent, symmetry, magical, sharp focus, open intriguing, inspired, new, shiny, amazing, creative, colorful, infinite, cute, artistic, passionate, pretty[Fooocus] Preparing Fooocus text #4 ...
[Prompt Expansion] the cat in the hat, bright, still, full perfect, detailed, highly saturated colors, cinematic, beautiful composition, dramatic, intricate, elegant, thought, epic, sharp focus, illuminated background, professional, amazing, creative, stunning, attractive, cute, poetic, pretty, best, awesome, romantic, pure, light, atmosphere, magic, ambient, pristine, extremely
[Fooocus] Preparing Fooocus text #5 ...
[Prompt Expansion] the cat in the hat, bright, still, dynamic composition, dramatic, cinematic, detailed, intricate, elegant, highly detail, professional focused, rich clear colorful background, beautiful, pure colors, light, inspired, designed, new classic, relaxed, confident, amazing, creative, positive, unique, attractive, best, atmosphere, pleasing, perfect, color, balance, symmetry
[Fooocus] Encoding positive #1 ...
[Fooocus] Encoding positive #2 ...
[Fooocus] Encoding positive #3 ...
[Fooocus] Encoding positive #4 ...
[Fooocus] Encoding positive #5 ...
[Fooocus] Encoding negative #1 ...
[Fooocus] Encoding negative #2 ...
[Fooocus] Encoding negative #3 ...
[Fooocus] Encoding negative #4 ...
[Fooocus] Encoding negative #5 ...
[Parameters] Denoising Strength = 1.0
[Parameters] Initial Latent shape: Image Space (1024, 1024)
Preparation time: 37.27 seconds
Using karras scheduler.
[Fooocus] GPU memory: max_reserved=0.000GB, max_allocated=0.000GB, reserved=0.000GB, free=3.221GB, free_torch=0.000GB, free_total=3.221GB, gpu_total=4.000GB, torch_total=0.000GB
[Fooocus] Preparing Fooocus task 1/5 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Requested to load SDXL
Loading 1 new model
loading in lowvram mode 1742.765775680542
[Fooocus Model Management] Moving model(s) has taken 7.76 seconds
  0%|                                                                                           | 0/30 [00:00<?, ?it/s]C:\SimpleAI\SimpleSDXL\modules\core.py:257: RuntimeWarning: invalid value encountered in cast
  x_sample = x_sample.cpu().numpy().clip(0, 255).astype(np.uint8)
 20%|████████████████▌                                                                  | 6/30 [04:40<18:41, 46.74s/it]
User stopped
[Enhance] Skipping, preconditions aren't met
Processing time (total): 288.82 seconds
Total time: 326.25 seconds
[Gallery] Refresh_output_catalog: loaded 322 images_catalogs.
[Gallery] Parse_html_log: loaded 1 image_infos of 24-08-20.
[Gallery] Refresh_images_catalog: loaded 1 image_items of 24-08-20.
[Gallery] Parse_html_log: loaded 1 image_infos of 24-08-20.

Additional information

This is the first time I have tried to make an image in SimpleSDXL. This problem occurs with both the juggernaut-X-RunDiffusion-NSFW.safetensors and RealvisXLv40_v40Bakedvae.safetensors base models.

I have a PC which does not meet the minimum requirements for mainline Fooocus. However, Fooocus has always run properly, it is just a bit slow. These are the PC specs:

Acer Nitro N50-600(DCH) CPU: Intel Core i5 9400F @2.90 GHz System RAM: 28 GB GPU: NVIDIA GeForce GTX 1650 with 4 GB VRAM

metercai commented 3 months ago

If the GPU memory is less than 6GB, it is recommended to still use Fooocus, as it is not possible to use the newly added models in SimpleSDXL.

DavidDragonsage commented 3 months ago

Unfortunately you gave me a different answer today at the mainline Fooocus site: https://github.com/lllyasviel/Fooocus/discussions/3548

While I do not have enough VRAM to run the non-SDXL models that require 6 GB or more, I was hoping to use the many other helpful features that SimpleSDXL has.

Interestingly enough, for some reason I was able to generate an image successfully when I used the SuperPrompt option (I found this out just now).

metercai commented 3 months ago

The main difference between SimpleSDXL and Fooocus is the addition of new models. I thought you were going to use those new models. Then I need to adapt to low VRM and block the functionality of the newly added models, which will be much better.

DavidDragonsage commented 3 months ago

Thank you, that will be wonderful! I see you have already released an update. I saw it detecting low VRAM and turning off Comfy.

Unfortunately I am still getting that same error once generation starts, as shown in the console log above:

0/30 [00:00<?, ?it/s]C:\SimpleAI\SimpleSDXL\modules\core.py:257: RuntimeWarning: invalid value encountered in cast x_sample = x_sample.cpu().numpy().clip(0, 255).astype(np.uint8)

There are several other features besides the new models that I am excited about, such as Super Prompt and IC-Light.

I have a small but active Facebook group, and there is a lot of enthusiasm for SimpleSDXL. I am keeping them updated on my progress. Someone said this new surprise is like Christmas! 🙂 https://www.facebook.com/groups/fooocus

DavidDragonsage commented 3 months ago

I don't know if this information may be helpful in troubleshooting this error, but I found these references to the error "invalid value encountered in cast x_sample =" https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/15412 https://github.com/lllyasviel/stable-diffusion-webui-forge/issues/1278

I did try the Euler/Normal combination of sampler and scheduler but I got the same error.

DavidDragonsage commented 2 months ago

I installed a 12GB VRAM card today and so this issue no longer affects me. However, if you need someone to test a possible fix I do not mind installing my old 4GB VRAM card to do so.

On the other hand, I now know that two other people with the NVIDIA GeForce GTX 1650 with 4 GB VRAM have been able to run both SDXL and Flux models in SimpleSDXL. And yesterday I was successful in running a Flux model with my old video card.

So I am thinking that my particular system was an anomaly, and of course the NVIDIA GeForce GTX 1650 with 4 GB VRAM was never approved for running mainline Fooocus. It may be best to close this bug report as being resolved!

metercai / SimpleSDXL