search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
16.96k stars 1.95k forks source link

[Bug]: reference image + batch size 4 = different results than batch size 1 #2183

Open dtxn opened 12 months ago

dtxn commented 12 months ago

Is there an existing issue for this?

What happened?

When I use reference_adain with default batch size 1 and repeat image generation with the same seed I get always the same result as it suppose to be. When I use reference_adain with default batch size 4 and then select the seed I liked from those 4 generated images and try to repeat generation, I get different results. They are not drastically different, but esthetically often not good (different shape mountain, instead of normal trees only sticks and so on). P.S. If to repeat the same without Controlnet enabled, then everything works as suppose to be.

Steps to reproduce the problem

  1. Enable controlnet with reference_adain, add random picture as reference image
  2. Select batch size of 4 in Auto1111
  3. Generate 4 images
  4. Select 1 image you like from those 4 and repeat the same seed of that liked image
  5. Generate
  6. Get different picture than it was rendered in batch 4

What should have happened?

Suppose to be the same result in batch 1 with the same seed as in batch 4 generated images.

Commit where the problem happens

webui: Auto1111 controlnet: v1.1.410

What browsers do you use to access the UI ?

No response

Command Line Arguments

No

List of enabled extensions

ControlNet v1.1.410

Console logs

Initializing Dreambooth
Dreambooth revision: 1a1d1621086a4725fda1200256f319c845dc7a8a
Successfully installed accelerate-0.23.0 fastapi-0.94.1 transformers-4.32.1

[!] xformers version 0.0.20 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.23.0 installed.
[+] diffusers version 0.21.4 installed.
[+] transformers version 4.32.1 installed.
[+] bitsandbytes version 0.41.1 installed.
Installing pyqt5 requirement for depthmap script
Launching Web UI with arguments: --xformers
[-] ADetailer initialized. version: 23.10.0, num models: 9
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
2023-10-19 22:11:44,878 - ControlNet - INFO - ControlNet v1.1.410
ControlNet preprocessor location: C:\Users\drais\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-10-19 22:11:45,181 - ControlNet - INFO - ControlNet v1.1.410
Loading weights [ec41bd2a82] from C:\Users\drais\stable-diffusion-webui\models\Stable-diffusion\photon_v1.safetensors
*Deforum ControlNet support: enabled*
Creating model from config: C:\Users\drais\stable-diffusion-webui\configs\v1-inference.yaml
False

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary C:\Users\drais\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary C:\Users\drais\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary C:\Users\drais\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary C:\Users\drais\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Loading VAE weights specified in settings: C:\Users\drais\stable-diffusion-webui\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 108.2s (prepare environment: 65.1s, import torch: 6.1s, import gradio: 1.4s, setup paths: 3.8s, initialize shared: 0.4s, other imports: 3.6s, setup codeformer: 0.6s, load scripts: 11.6s, create ui: 11.1s, gradio launch: 4.1s).Applying attention optimization: xformers... done.
Model loaded in 18.9s (load weights from disk: 2.0s, create model: 1.2s, apply weights to model: 7.9s, apply half(): 0.9s, load VAE: 4.3s, calculate empty prompt: 2.5s).
Restoring base VAE
Applying attention optimization: xformers... done.
VAE weights loaded.
Loading VAE weights specified in settings: C:\Users\drais\stable-diffusion-webui\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors
Applying attention optimization: xformers... done.
VAE weights loaded.
2023-10-19 22:17:15,671 - ControlNet - INFO - Loading preprocessor: reference_adain
2023-10-19 22:17:15,672 - ControlNet - INFO - preprocessor resolution = 640
2023-10-19 22:17:16,243 - ControlNet - INFO - ControlNet Hooked - Time = 0.9826855659484863
  0%|                                                                                           | 0/30 [00:00<?, ?it/s]2023-10-19 22:17:18,591 - ControlNet - INFO - ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 80, 192]).
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:18<00:00,  1.59it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 30/30 [00:17<00:00,  1.76it/s]
2023-10-19 22:18:29,864 - ControlNet - INFO - Loading preprocessor: reference_adain████| 30/30 [00:17<00:00,  1.82it/s]
2023-10-19 22:18:29,876 - ControlNet - INFO - preprocessor resolution = 640
2023-10-19 22:18:30,371 - ControlNet - INFO - ControlNet Hooked - Time = 0.5238332748413086

Additional information

None

huchenlei commented 12 months ago

In fact, generate with same seed batch size 1 would also not produce the exact same result(pixel by pixel) with reference mode.

See https://github.com/Mikubill/sd-webui-controlnet/blob/a43e574254d19a362082bbd412f24aeef1beed47/web_tests/main.py#L246

It is a known issue also for shuffle. Probably ask lvmin for a fix as I know little where does this nondeterministic behavior come from.

dtxn commented 12 months ago

In fact, generate with same seed batch size 1 would also not produce the exact same result(pixel by pixel) with reference mode.

See

https://github.com/Mikubill/sd-webui-controlnet/blob/a43e574254d19a362082bbd412f24aeef1beed47/web_tests/main.py#L246

It is a known issue also for shuffle. Probably ask lvmin for a fix as I know little where does this nondeterministic behavior come from.

Hi. I understand what you mean "pixel by pixel". It's natural that there are minor changes- added or removed some details, but usually you can find that when already compare image to image. I was talking, that the difference is 10-20%. Let's say in batch 4 it was a picture with nice trees on the rock, but when regenerated the same seed but already only batch size 1, instead of trees I got a house on the rock.

huchenlei commented 11 months ago

@lllyasviel Do you have any idea why this happens?

My guess is that the difference on initial seed build up during the same batch, which makes the generation results towards the tail of batch having more variations. If that is the case, there does not seem to be much we can do.