[Bug]: Reference mode VERY dark

[alenknight]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

whenever I run reference mode, the resulting image is INSANELY dark. and often very very saturated. tried on a few computers.

Steps to reproduce the problem

1. Go to .... controlnet
2. add an image
3. set to reference only
4. basic prompt. my example is longer but any prompt just the words "test" works
5. render
6. image is very very dark and hyper saturated

I'm running on nvidia cards. but I've tried on 2 computers. one with a 3060 ti and one with a 3090. issue same. some models are slightly better. but for the most part it's the same issue.

What should have happened?

image shouldn't be dark and saturated

Commit where the problem happens

webui: https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/baf6946e06249c5af9851c60171692c44ef633e0 controlnet: https://github.com/Mikubill/sd-webui-controlnet/commit/0b7f43107ccc7eb3b98b224db1407baec5f4b4f3

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome, Apple Safari, Microsoft Edge

Command Line Arguments

--listen --port 7777 --opt-split-attention --disable-nan-check --api --enable-insecure-extension-access

List of enabled extensions

Console logs

venv "C:\AI\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.3.2
Commit hash: baf6946e06249c5af9851c60171692c44ef633e0
Installing requirements

current transparent-background 1.2.4

Requirement already satisfied: send2trash~=1.8 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (1.8.2)
Requirement already satisfied: dynamicprompts[attentiongrabber,magicprompt]~=0.27.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (0.27.0)
Requirement already satisfied: pyparsing~=3.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.1.0)
Requirement already satisfied: jinja2~=3.1 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.1.2)
Requirement already satisfied: transformers[torch]~=4.19 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (4.25.1)
Requirement already satisfied: MarkupSafe>=2.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from jinja2~=3.1->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2.1.3)
Requirement already satisfied: tqdm>=4.27 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (4.65.0)
Requirement already satisfied: filelock in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.12.2)
Requirement already satisfied: regex!=2019.12.17 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2023.6.3)
Requirement already satisfied: pyyaml>=5.1 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (6.0)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (0.13.3)
Requirement already satisfied: requests in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2.31.0)
Requirement already satisfied: packaging>=20.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (23.1)
Requirement already satisfied: huggingface-hub<1.0,>=0.10.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (0.15.1)
Requirement already satisfied: numpy>=1.17 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (1.23.5)
Requirement already satisfied: torch!=1.12.0,>=1.7 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2.0.1+cu118)
Requirement already satisfied: fsspec in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from huggingface-hub<1.0,>=0.10.0->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2023.6.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from huggingface-hub<1.0,>=0.10.0->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (4.6.3)
Requirement already satisfied: sympy in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from torch!=1.12.0,>=1.7->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (1.12)
Requirement already satisfied: networkx in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from torch!=1.12.0,>=1.7->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.1)
Requirement already satisfied: colorama in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from tqdm>=4.27->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (0.4.6)
Requirement already satisfied: idna<4,>=2.5 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from requests->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from requests->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from requests->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2023.5.7)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from requests->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.1.0)
Requirement already satisfied: mpmath>=0.19 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from sympy->torch!=1.12.0,>=1.7->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (1.3.0)
sd-dynamic-prompts installer: running 'C:\AI\stable-diffusion-webui\venv\Scripts\python.exe' -m pip install 'send2trash~=1.8' 'dynamicprompts[attentiongrabber,magicprompt]~=0.27.0'

Installing opencv-python
Installing opencv-python for pixel extension

Checking roop requirements
Install insightface==0.7.3
Installing sd-webui-roop requirement: insightface==0.7.3
Install onnx==1.14.0
Installing sd-webui-roop requirement: onnx==1.14.0
Install onnxruntime==1.15.0
Installing sd-webui-roop requirement: onnxruntime==1.15.0
Install opencv-python==4.7.0.72
Installing sd-webui-roop requirement: opencv-python==4.7.0.72

If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: dc413a14379b165355502d9f65856c40a4bb5b6f
Successfully installed accelerate-0.19.0 fastapi-0.94.1 gitpython-3.1.31 transformers-4.29.2

Does your project take forever to startup?
Repetitive dependency installation may be the reason.
Automatic1111's base project sets strict requirements on outdated dependencies.
If an extension is using a newer version, the dependency is uninstalled and reinstalled twice every startup.

[+] xformers version 0.0.17 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.19.0 installed.
[+] diffusers version 0.16.1 installed.
[+] transformers version 4.29.2 installed.
[+] bitsandbytes version 0.35.4 installed.

Launching Web UI with arguments: --listen --port 7777 --opt-split-attention --disable-nan-check --api --enable-insecure-extension-access
No module 'xformers'. Proceeding without it.
[-] ADetailer initialized. version: 23.6.3, num models: 11
2023-06-24 15:37:41,654 - ControlNet - INFO - ControlNet v1.1.227
ControlNet preprocessor location: C:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-06-24 15:37:41,962 - ControlNet - INFO - ControlNet v1.1.227
Loading weights [098e338b2f] from C:\AI\stable-diffusion-webui\models\Stable-diffusion\fantasy\Lunar-Diffusion-1.29.safetensors
Creating model from config: C:\AI\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Textual inversion embeddings loaded(120): advntr, angry512, anime512, ansm1th_ananicolesmith, AS-Adult, AS-Adult-neg, AS-Elderly, AS-MidAged, AS-Young, AS-Young-neg, AS-Younger, AS-YoungerV2, AS-Youngest, AS-YoungestV2, AS-YoungV2, AS-YoungV2-neg, Asian-Less, Asian-Less-Neg, Asian-Less-Toon, AuroraNegative, awful512, bad-artist, bad-artist-anime, bad-hands-5, bad-image-9600, bad-image-v2-11000, bad-image-v2-27000, bad-image-v2-39000, bad-picture-chill-1v, bad-picture-chill-32v, bad-picture-chill-75v, bad_prompt, bad_prompt_version2, bad_prompt_version2-neg, BadDream, badhandsv5-neg, badhandv4, bazaldua_v2-10000, beauty512, boring_e621, corneo_covering_breasts_two_hands, defiance512, dichen, easynegative, emb-anada, emb-avril, emb-rrf2, emmastone, EmWat69, FastNegativeEmbedding, FastNegativeEmbeddingStrong, fcDetailPortrait, fcHeatPortrait, fcNeg-neg, fcPortrait, grin512, GS-Boyish, GS-DeFeminize-neg, GS-DeMasculate-neg, GS-Masculine, happy512, HAROQUE, HRGiger, HRGigerEmb, is, JuggernautNegative-neg, kaleidcp-3450, KHFB, laugh512, linds3yp3las, lnrneg, lnrpos, lr, MLKEmb, n3t0p, neg_anime512, neg_facelift512, neg_realism512, negative_hand-neg, nervous512, ng_deepnegative_v1_16t, ng_deepnegative_v1_2t, ng_deepnegative_v1_32t, ng_deepnegative_v1_4t, ng_deepnegative_v1_64t, ng_deepnegative_v1_75t, nRealisticVision, opt-6000, Portrait512, pureerosface_v1, realisticvision-negative-embedding, render512, rfktr_bontrex-150, rfktr_bwmnga, rfktr_circrex, RFKTR_fshnsta, RFKTR_plastic, RFKTR_rootrex, rfktr_sinetempore, rmadanegative402_sd15-neg, sad512, sagbobz, selfie512, shock512, smile512, Style-Empire, Style-Empire-neg, style-hamunaptra, style-psycho, style-widow, suckhertonguealexV2.0, suckhertonguealexzuov4, suckhertonguealexzuov5, sxz-raiden-mgr, the_trump, ulzzang-6500, ulzzang-6500-v1.1, UnrealisticDream, wholesomegrandpas, wholesomegrannies
Textual inversion embeddings skipped(23): AnalogFilm768-BW-Classic, AnalogFilm768-BW-Modern, AnalogFilm768-BW-Tintype, AnalogFilm768-BW-Vintage, AnalogFilm768-Old-School, AnalogFilm768, Cinema768-Analog, Cinema768-BW, Cinema768-Classic, Cinema768-SilentFilm, FloralMarble-150, FloralMarble-250, FloralMarble-400, FloralMarble, nartfixer, Neg_Facelift768, nfixer, no_unrealistic768, nrealfixer, style-hamunaptra_sd2, style-widow_sd2, TungstenDispo, ZphyrInsanity_Negative-neg
Model loaded in 1.9s (load weights from disk: 0.2s, create model: 0.4s, apply weights to model: 0.4s, apply half(): 0.3s, move model to device: 0.5s, load textual inversion embeddings: 0.2s).
2023-06-24 15:37:47,880 - roop - INFO - roop v0.0.2
2023-06-24 15:37:47,881 - roop - INFO - roop v0.0.2
Image Browser: ImageReward is not installed, cannot be used.
Applying optimization: Doggettx... done.
CUDA SETUP: Loading binary C:\AI\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cudaall.dll...
Running on local URL:  http://0.0.0.0:7777

To create a public link, set `share=True` in `launch()`.
Startup time: 21.7s (import torch: 1.8s, import gradio: 1.4s, import ldm: 0.4s, other imports: 0.9s, setup codeformer: 0.2s, list SD models: 0.4s, load scripts: 10.4s, create ui: 1.7s, gradio launch: 4.3s).
2023-06-24 15:42:20,725 - ControlNet - INFO - Loading preprocessor: reference_only
2023-06-24 15:42:20,725 - ControlNet - INFO - preprocessor resolution = 543
  0%|                                                                                           | 0/50 [00:00<?, ?it/s]2023-06-24 15:42:28,433 - ControlNet - INFO - ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 96, 64]).
100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:20<00:00,  2.39it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:19<00:00,  2.63it/s]

Additional information

issue seems repeatable, but I also saw this in past versions too. but it wasn't as bad. I've also tried reinstalling A1111 with a fresh install of controlnet and NO other extensions. same issue.

[alenknight]

I should have added examples. this prompt

[(a medium shot photo of a masculine Japanese (Samurai:1.2) warrior, tattooed robot,  samurai armor , helmet, 
art by Ayami Kojima and Naoto Hattori, featured on zbrush central
<lora:samuraiLoraV01:.35><lora:insanobot:.25>):
((hero shot of a man wearing a (batman:1.2) costume), (Samurai armor:1.4),
(metal Bronze batman symbol chest:1.1)):.39], 
ornate armor, (HRgiger:1.1),
(photorealistic:1.5), raining village background,(mist:0.2), highest quality, particles, light rain, man focus, muscular shoulders, night, outdoors, (armored costume:1.5), (mask:1.2), rain, dark atmosphere, lightning, detailed background, masterpiece, moody, (realistic:1.3),  perfect face, perfect hands, perfect lighting, perfect shading, (realistic fabric texture), volumetric shading, subsurface scattering, hero pose, dynamic pose, dynamic movement, 
japanese ancient village background, (raining:1.2), (looking at the camera:1.2), 
gigachad, biomechanical, hyper realistic, hyper detailed, intricate, insane fine details, Extremely sharp lines, photorealistic, Key Light Setup, Canon EF 85mm f/1.2L II USM,
[batman:samurai]
<lora:lowra_v10:.45> <lora:epiNoiseoffset_v2:.75><lora:add_detail:1> 

Negative:(bad_prompt_version2:0.7), (bad-hands-5:0.7), (greyscale, monochrome:1.1), BadDream, (UnrealisticDream:1.2), negative_hand-neg, EasyNegativeV2, (deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime:1.4), close up, cropped, out of frame, extra frame, lowres, worst quality, low quality, normal quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arm, missing leg, bare arms, extra legs, long neck, (extra fingers, deformed fingers, missing fingers, fused fingers, too many fingers, deformed hands, mutated hands, poorly drawn hands:1.4), text, error, extra digit, fewer digits, signature, watermark, username, artist name, deformed, blur, blurred, macro, (skinny waist:1.25)

with this controlnet image in reference mode:

produces this image

[huchenlei]

Not reproducable with the steps you provided:

I probably don't have the LoRAs and embeddings in your prompt. Can you check whether the issue persists with no prompt / simple prompt?

[(a medium shot photo of a masculine Japanese (Samurai:1.2) warrior, tattooed robot, samurai armor , helmet,
art by Ayami Kojima and Naoto Hattori, featured on zbrush central
<lora:samuraiLoraV01:.35><lora:insanobot:.25>):
((hero shot of a man wearing a (batman:1.2) costume), (Samurai armor:1.4),
(metal Bronze batman symbol chest:1.1)):.39],
ornate armor, (HRgiger:1.1),
(photorealistic:1.5), raining village background,(mist:0.2), highest quality, particles, light rain, man focus, muscular shoulders, night, outdoors, (armored costume:1.5), (mask:1.2), rain, dark atmosphere, lightning, detailed background, masterpiece, moody, (realistic:1.3), perfect face, perfect hands, perfect lighting, perfect shading, (realistic fabric texture), volumetric shading, subsurface scattering, hero pose, dynamic pose, dynamic movement,
japanese ancient village background, (raining:1.2), (looking at the camera:1.2),
gigachad, biomechanical, hyper realistic, hyper detailed, intricate, insane fine details, Extremely sharp lines, photorealistic, Key Light Setup, Canon EF 85mm f/1.2L II USM,
[batman:samurai]
<lora:lowra_v10:.45> <lora:epiNoiseoffset_v2:.75><lora:add_detail:1>
Negative prompt: (bad_prompt_version2:0.7), (bad-hands-5:0.7), (greyscale, monochrome:1.1), BadDream, (UnrealisticDream:1.2), negative_hand-neg, EasyNegativeV2, (deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime:1.4), close up, cropped, out of frame, extra frame, lowres, worst quality, low quality, normal quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arm, missing leg, bare arms, extra legs, long neck, (extra fingers, deformed fingers, missing fingers, fused fingers, too many fingers, deformed hands, mutated hands, poorly drawn hands:1.4), text, error, extra digit, fewer digits, signature, watermark, username, artist name, deformed, blur, blurred, macro, (skinny waist:1.25)
Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 9, Seed: 3201975704, Size: 512x768, Model hash: b42b09ff12, Model: cetusMix_v4, ControlNet 1: "preprocessor: reference_only, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, 0.5, 2.7)", Version: v1.3.2-1-g56a253d3

Failed to find Loras: samuraiLoraV01, insanobot, lowra_v10, epiNoiseoffset_v2, add_detail
Used embeddings: negative_hand-neg [b740]

[alenknight]

Interesting!!!! yes it seems like it's specifically these 2 LORA's.... odd cause I swear I remember trying this without lora's before... but yeah when I take these out... it works just fine!

I know these by default increase the black levels and tonal range... but odd how they mixing to result in such a black image... when it doesn't happen when using other controlnet models. still... guess that solves it. it's not a Controlnet issue per-se.... still... sure wish I could understand better WHY these specific lora's cause this. (more so I can figure out about training Loras' better)

[qidai77777]

I have also encountered this problem, I do not use any lora

[qidai77777]