[Bug]: Hires. fix causes clones at every setting

JoyBell2 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits

What happened?

Hires fix causes clones at all possible settings and options, if a prompt has two or more characters they will all end up cloned after hires.fix is applyed. Grid of all options Cloning from HD Fix Image with no Hires.fix 00227-3269171256

Second example 00224-4104266214 photograph of Anne Hathaway dancing with Ryan Reynolds at a Brazilian night club, well dressed up for a night out, by Vogue Magazine epiCRealism Negative prompt: epiCNegative, nude Steps: 50, Sampler: DPM++ SDE Karras, CFG scale: 7, Seed: 4104266214, Size: 960x640, Model hash: 980cb713af, Model: reliberate_v10, TI hashes: "epiCRealism: 3719288307e4", Version: v1.5.0

Simple Beautiful. Well done. Hires fix... two Anne Hathaways! 00226-4104266214

This happens with any prompt and all tried settings and even None and 0.0 Denoise. If there are two or more people we all get morphed together or the one named person.

Steps to reproduce the problem

Make a prompt with two distinct faces.
Use Hires.fix at any settings you choose.
Get unfixable clones.

What should have happened?

We darn well should have Anne Hathaway dancing with Ryan Renolds. Enough said.

Version or Commit where the problem happens

version: v1.5.0

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

Cross attention optimization

xformers

What browsers do you use to access the UI ?

Microsoft Edge

Command Line Arguments

git pull
@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= --xformers --opt-split-attention --medvram
call webui.bat

List of extensions

Extension | URL | Branch | Version | Date | Update -- | -- | -- | -- | -- | -- ABG_extension | https://github.com/KutsuyaYuki/ABG_extension | main | eb3ff761 | Mon May 1 21:25:47 2023 | unknown Stable-Diffusion-Webui-Civitai-Helper | https://github.com/butaixianran/Stable-Diffusion-Webui-Civitai-Helper.git | main | 920ca326 | Tue May 23 11:53:22 2023 | unknown a1111-sd-webui-lycoris | https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris | main | 8e97bf54 | Sun Jul 9 07:44:58 2023 | unknown ddetailer | https://github.com/dustysys/ddetailer | master | 22a17414 | Mon Nov 28 09:33:31 2022 | unknown loopback_scaler | https://github.com/Elldreth/loopback_scaler.git | main | 184a53e6 | Fri Jun 23 21:42:22 2023 | unknown multi-subject-render | https://github.com/Extraltodeus/multi-subject-render.git | main | 03427e26 | Mon Mar 6 14:11:30 2023 | unknown multidiffusion-upscaler-for-automatic1111 | https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111.git | main | f9f8073e | Sat Jul 22 06:53:11 2023 | unknown sd-model-organizer | https://github.com/alexandersokol/sd-model-organizer.git | main | d6a2e6af | Mon Jul 24 22:18:01 2023 | unknown sd-model-preview-xd | https://github.com/CurtisDS/sd-model-preview-xd | main | 59bcf88f | Sat Jul 22 12:30:04 2023 | unknown sd-webui-llul | https://github.com/hnmr293/sd-webui-llul.git | master | aa47b3ee | Thu May 4 16:14:34 2023 | unknown sd-webui-stablesr | https://github.com/pkuliyi2015/sd-webui-stablesr.git | master | 4499d796 | Sat Jul 1 16:38:32 2023 | unknown stable-diffusion-webui-anti-burn | https://github.com/klimaleksus/stable-diffusion-webui-anti-burn.git | master | 4d678f1f | Sun Feb 19 19:22:44 2023 | unknown stable-diffusion-webui-eyemask | https://github.com/ilian6806/stable-diffusion-webui-eyemask.git | master | 7b803a43 | Fri Jun 2 11:15:19 2023 | unknown stable-diffusion-webui-two-shot | https://github.com/opparco/stable-diffusion-webui-two-shot | main | 9936c52e | Sun Feb 19 08:40:41 2023 | unknown ultimate-upscale-for-automatic1111 | https://github.com/Coyote-A/ultimate-upscale-for-automatic1111 | master | c99f382b | Tue Jun 13 04:29:35 2023 | unknown LDSR | built-in | None | | Wed Jul 26 23:59:05 2023 | Lora | built-in | None | | Wed Jul 26 23:59:05 2023 | ScuNET | built-in | None | | Wed Jul 26 23:59:05 2023 | SwinIR | built-in | None | | Wed Jul 26 23:59:05 2023 | canvas-zoom-and-pan | built-in | None | | Wed Jul 26 23:59:05 2023 | extra-options-section | built-in | None | | Wed Jul 26 23:59:05 2023 | mobile | built-in | None | | Wed Jul 26 23:59:05 2023 | prompt-bracket-checker | built-in | None | | Wed Jul 26 23:59:05 2023 | Extension URL Branch Version Date Update ABG_extension https://github.com/KutsuyaYuki/ABG_extension main [eb3ff761](https://github.com/KutsuyaYuki/ABG_extension/commit/eb3ff7610abeca1e4941a398b85aff223e32fba3) Mon May 1 21:25:47 2023 unknown Stable-Diffusion-Webui-Civitai-Helper https://github.com/butaixianran/Stable-Diffusion-Webui-Civitai-Helper.git main [920ca326](https://github.com/butaixianran/Stable-Diffusion-Webui-Civitai-Helper/commit/920ca3267f789a72e3225c99f069bf0be0db795d) Tue May 23 11:53:22 2023 unknown a1111-sd-webui-lycoris https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris main [8e97bf54](https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris/commit/8e97bf54867c25d00fc480be1ab4dae5399b35ef) Sun Jul 9 07:44:58 2023 unknown ddetailer https://github.com/dustysys/ddetailer master [22a17414](https://github.com/dustysys/ddetailer/commit/22a17414bbca1fa4681d427b8aae5f970d17e896) Mon Nov 28 09:33:31 2022 unknown loopback_scaler https://github.com/Elldreth/loopback_scaler.git main [184a53e6](https://github.com/Elldreth/loopback_scaler/commit/184a53e6d22579c578c6ee1bf55d821479213c49) Fri Jun 23 21:42:22 2023 unknown multi-subject-render https://github.com/Extraltodeus/multi-subject-render.git main [03427e26](https://github.com/Extraltodeus/multi-subject-render/commit/03427e26bebdc6da0ccfb749bf3c4e7e33d7458b) Mon Mar 6 14:11:30 2023 unknown multidiffusion-upscaler-for-automatic1111 https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111.git main [f9f8073e](https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111/commit/f9f8073e64f4e682838f255215039ba7884553bf) Sat Jul 22 06:53:11 2023 unknown sd-model-organizer https://github.com/alexandersokol/sd-model-organizer.git main [d6a2e6af](https://github.com/alexandersokol/sd-model-organizer/commit/d6a2e6af54d049db6741b0990c999ebca1b98f31) Mon Jul 24 22:18:01 2023 unknown sd-model-preview-xd https://github.com/CurtisDS/sd-model-preview-xd main [59bcf88f](https://github.com/CurtisDS/sd-model-preview-xd/commit/59bcf88f4763dc1864463e6ac80859e549c089ce) Sat Jul 22 12:30:04 2023 unknown sd-webui-llul https://github.com/hnmr293/sd-webui-llul.git master [aa47b3ee](https://github.com/hnmr293/sd-webui-llul/commit/aa47b3eeb45c53f0d6ccaae59abf36e8ed6731f5) Thu May 4 16:14:34 2023 unknown sd-webui-stablesr https://github.com/pkuliyi2015/sd-webui-stablesr.git master [4499d796](https://github.com/pkuliyi2015/sd-webui-stablesr/commit/4499d79649e54ca27b196e8e55a65d2a64a1b75c) Sat Jul 1 16:38:32 2023 unknown stable-diffusion-webui-anti-burn https://github.com/klimaleksus/stable-diffusion-webui-anti-burn.git master [4d678f1f](https://github.com/klimaleksus/stable-diffusion-webui-anti-burn/commit/4d678f1f1120415fe4cb9f77484252bc82af03b2) Sun Feb 19 19:22:44 2023 unknown stable-diffusion-webui-eyemask https://github.com/ilian6806/stable-diffusion-webui-eyemask.git master [7b803a43](https://github.com/ilian6806/stable-diffusion-webui-eyemask/commit/7b803a438f02b381903916b65cc5b15c98e3231c) Fri Jun 2 11:15:19 2023 unknown stable-diffusion-webui-two-shot https://github.com/opparco/stable-diffusion-webui-two-shot main [9936c52e](https://github.com/opparco/stable-diffusion-webui-two-shot/commit/9936c52eba04da7f0c009051ab03a60e72d4f7e6) Sun Feb 19 08:40:41 2023 unknown ultimate-upscale-for-automatic1111 https://github.com/Coyote-A/ultimate-upscale-for-automatic1111 master [c99f382b](https://github.com/Coyote-A/ultimate-upscale-for-automatic1111/commit/c99f382b31509b87b4d512e70e9caf08ae7a079f) Tue Jun 13 04:29:35 2023 unknown LDSR [built-in](http://127.0.0.1:7860/) None Wed Jul 26 23:59:05 2023 Lora [built-in](http://127.0.0.1:7860/) None Wed Jul 26 23:59:05 2023 ScuNET [built-in](http://127.0.0.1:7860/) None Wed Jul 26 23:59:05 2023 SwinIR [built-in](http://127.0.0.1:7860/) None Wed Jul 26 23:59:05 2023 canvas-zoom-and-pan [built-in](http://127.0.0.1:7860/) None Wed Jul 26 23:59:05 2023 extra-options-section [built-in](http://127.0.0.1:7860/) None Wed Jul 26 23:59:05 2023 mobile [built-in](http://127.0.0.1:7860/) None Wed Jul 26 23:59:05 2023 prompt-bracket-checker [built-in](http://127.0.0.1:7860/) None Wed Jul 26 23:59:05 2023 ### Console logs ```Shell C:\Stable Diffusion 3\stable-diffusion-webui>git pull Already up to date. venv "C:\Stable Diffusion 3\stable-diffusion-webui\venv\Scripts\Python.exe" Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] Version: v1.5.0 Commit hash: a3ddf464a2ed24c999f67ddfef7969f8291567be Installing dlib Couldn't install dlib. Command: "C:\Stable Diffusion 3\stable-diffusion-webui\venv\Scripts\python.exe" -m pip install dlib==19.24.0 Error code: 1 stdout: Collecting dlib==19.24.0 Using cached dlib-19.24.0.tar.gz (3.2 MB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Building wheels for collected packages: dlib Building wheel for dlib (setup.py): started Building wheel for dlib (setup.py): finished with status 'error' Running setup.py clean for dlib Failed to build dlib stderr: WARNING: Ignoring invalid distribution -rotobuf (c:\stable diffusion 3\stable-diffusion-webui\venv\lib\site-packages) error: subprocess-exited-with-error python setup.py bdist_wheel did not run successfully. exit code: 1 [7 lines of output] running bdist_wheel running build running build_py running build_ext ERROR: CMake must be installed to build dlib [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for dlib ERROR: Could not build wheels for dlib, which is required to install pyproject.toml-based projects [notice] A new release of pip is available: 23.1.2 -> 23.2.1 [notice] To update, run: C:\Stable Diffusion 3\stable-diffusion-webui\venv\Scripts\python.exe -m pip install --upgrade pip ---------------------------------------------- Failed building wheel for dlib ERROR: CMake must be installed to build dlib Install cmake from https://cmake.org/download/ ---------------------------------------------- Launching Web UI with arguments: --xformers --opt-split-attention --medvram Civitai Helper: Get Custom Model Folder Civitai Helper: Load setting from: C:\Stable Diffusion 3\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\setting.json Civitai Helper: No setting file, use default Installing openmim Installing mmcv-full *** Error loading script: ddetailer.py Traceback (most recent call last): File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\scripts.py", line 295, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module module_spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\ddetailer\scripts\ddetailer.py", line 61, in startup() File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\ddetailer\scripts\ddetailer.py", line 49, in startup run(f'"{python}" -m mim install mmcv-full', desc=f"Installing mmcv-full", errdesc=f"Couldn't install mmcv-full") File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\launch_utils.py", line 113, in run raise RuntimeError("\n".join(error_bits)) RuntimeError: Couldn't install mmcv-full. Command: "C:\Stable Diffusion 3\stable-diffusion-webui\venv\Scripts\python.exe" -m mim install mmcv-full Error code: 1 stdout: Looking in links: https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html Collecting mmcv-full Using cached mmcv-full-1.7.1.tar.gz (605 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error' stderr: WARNING: Ignoring invalid distribution -rotobuf (c:\stable diffusion 3\stable-diffusion-webui\venv\lib\site-packages) error: subprocess-exited-with-error python setup.py egg_info did not run successfully. exit code: 1 [14 lines of output] Traceback (most recent call last): File "", line 2, in File "", line 34, in File "C:\Users\mcmun\AppData\Local\Temp\pip-install-utwbtcm_\mmcv-full_73a373ed32974d3bbb7dee40e59a28ea\setup.py", line 465, in ext_modules=get_extensions(), File "C:\Users\mcmun\AppData\Local\Temp\pip-install-utwbtcm_\mmcv-full_73a373ed32974d3bbb7dee40e59a28ea\setup.py", line 369, in get_extensions ext_ops = extension( File "C:\Stable Diffusion 3\stable-diffusion-webui\venv\lib\site-packages\torch\utils\cpp_extension.py", line 1048, in CUDAExtension library_dirs += library_paths(cuda=True) File "C:\Stable Diffusion 3\stable-diffusion-webui\venv\lib\site-packages\torch\utils\cpp_extension.py", line 1186, in library_paths paths.append(_join_cuda_home(lib_dir)) File "C:\Stable Diffusion 3\stable-diffusion-webui\venv\lib\site-packages\torch\utils\cpp_extension.py", line 2223, in _join_cuda_home raise EnvironmentError('CUDA_HOME environment variable is not set. ' OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root. [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed Encountered error while generating package metadata. See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. [notice] A new release of pip is available: 23.1.2 -> 23.2.1 [notice] To update, run: C:\Stable Diffusion 3\stable-diffusion-webui\venv\Scripts\python.exe -m pip install --upgrade pip --- *** Error loading script: em_api.py Traceback (most recent call last): File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\scripts.py", line 295, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module module_spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\em_api.py", line 16, in from eyemask.api import EyeMaskApi File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\api\__init__.py", line 3, in from . api import EyeMaskApi File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\api\api.py", line 8, in from .. import script as eye_mask_script File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\script.py", line 7, in from . import mask_generator, utils, widlcards File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\mask_generator.py", line 2, in import dlib ModuleNotFoundError: No module named 'dlib' --- *** Error loading script: em_script.py Traceback (most recent call last): File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\scripts.py", line 295, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module module_spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\em_script.py", line 15, in from eyemask import constants, ui, script as eye_mask_script File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\ui.py", line 8, in from . import constants, script, script_embedded, utils, widlcards, state, mask_generator File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\script.py", line 7, in from . import mask_generator, utils, widlcards File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\mask_generator.py", line 2, in import dlib ModuleNotFoundError: No module named 'dlib' --- *** Error loading script: em_script_embedded.py Traceback (most recent call last): File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\scripts.py", line 295, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "C:\Stable Diffusion 3\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module module_spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\em_script_embedded.py", line 15, in from eyemask import constants, ui, script_embedded as eye_mask_script File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\ui.py", line 8, in from . import constants, script, script_embedded, utils, widlcards, state, mask_generator File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\script.py", line 7, in from . import mask_generator, utils, widlcards File "C:\Stable Diffusion 3\stable-diffusion-webui\extensions\stable-diffusion-webui-eyemask\scripts\eyemask\mask_generator.py", line 2, in import dlib ModuleNotFoundError: No module named 'dlib' --- Loading weights [0466ae65a0] from C:\Stable Diffusion 3\stable-diffusion-webui\models\Stable-diffusion\EpicDiffusion11.safetensors Creating model from config: C:\Stable Diffusion 3\stable-diffusion-webui\configs\v1-inference.yaml LatentDiffusion: Running in eps-prediction mode DiffusionWrapper has 859.52 M params. Applying attention optimization: xformers... done. Model loaded in 3.0s (load weights from disk: 0.5s, create model: 0.6s, apply weights to model: 0.6s, apply half(): 0.6s, calculate empty prompt: 0.7s). Running on local URL: http://127.0.0.1:7860 To create a public link, set `share=True` in `launch()`. Startup time: 37.3s (launcher: 16.1s, import torch: 2.8s, import gradio: 1.0s, setup paths: 0.6s, other imports: 0.8s, list SD models: 0.2s, load scripts: 12.3s, create ui: 3.2s, gradio launch: 0.1s). Loading weights [980cb713af] from C:\Stable Diffusion 3\stable-diffusion-webui\models\Stable-diffusion\reliberate_v10.safetensors Applying attention optimization: xformers... done. Weights loaded in 0.5s (load weights from disk: 0.1s, apply weights to model: 0.3s). 100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:19<00:00, 2.63it/s] [Tiled VAE]: the input size is tiny and unnecessary to tile.███████████████████████████| 50/50 [00:17<00:00, 2.79it/s] Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:18<00:00, 2.76it/s] 100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:17<00:00, 2.86it/s] [Tiled VAE]: the input size is tiny and unnecessary to tile.████████████████ | 50/60 [00:41<00:03, 2.94it/s] Tile 1/24 Tile 2/24 Tile 3/24 Tile 4/24 Tile 5/24 Tile 6/24 Tile 7/24 Tile 8/24 Tile 9/24 Tile 10/24 Tile 11/24 Tile 12/24 Tile 13/24 Tile 14/24 Tile 15/24 Tile 16/24 Tile 17/24 Tile 18/24 Tile 19/24 Tile 20/24 Tile 21/24 Tile 22/24 Tile 23/24 Tile 24/24 [Tiled VAE]: the input size is tiny and unnecessary to tile. 100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:19<00:00, 1.99s/it] [Tiled VAE]: input_size: torch.Size([1, 4, 160, 240]), tile_size: 192, padding: 11█████| 60/60 [01:05<00:00, 2.08s/it] [Tiled VAE]: split to 1x2 = 2 tiles. Optimal tile size 128x160, original tile size 192x192 [Tiled VAE]: Fast mode enabled, estimating group norm parameters on 192 x 128 image [Tiled VAE]: Executing Decoder Task Queue: 100%|████████████████████████████████████| 246/246 [00:00<00:00, 307.60it/s] [Tiled VAE]: Done in 1.820s, max VRAM alloc 3562.876 MB Total progress: 100%|██████████████████████████████████████████████████████████████████| 60/60 [01:08<00:00, 1.14s/it] 100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:17<00:00, 2.87it/s] [Tiled VAE]: the input size is tiny and unnecessary to tile.███████████████████████▎ | 50/53 [00:17<00:01, 2.95it/s] Tile 1/24 Tile 2/24 Tile 3/24 Tile 4/24 Tile 5/24 Tile 6/24 Tile 7/24 Tile 8/24 Tile 9/24 Tile 10/24 Tile 11/24 Tile 12/24 Tile 13/24 Tile 14/24 Tile 15/24 Tile 16/24 Tile 17/24 Tile 18/24 Tile 19/24 Tile 20/24 Tile 21/24 Tile 22/24 Tile 23/24 Tile 24/24 [Tiled VAE]: the input size is tiny and unnecessary to tile. 100%|████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:05<00:00, 1.83s/it] [Tiled VAE]: input_size: torch.Size([1, 4, 160, 240]), tile_size: 192, padding: 11█████| 53/53 [00:26<00:00, 1.84s/it] [Tiled VAE]: split to 1x2 = 2 tiles. Optimal tile size 128x160, original tile size 192x192 [Tiled VAE]: Fast mode enabled, estimating group norm parameters on 192 x 128 image [Tiled VAE]: Executing Decoder Task Queue: 100%|████████████████████████████████████| 246/246 [00:00<00:00, 330.65it/s] [Tiled VAE]: Done in 1.691s, max VRAM alloc 3562.876 MB Total progress: 100%|██████████████████████████████████████████████████████████████████| 53/53 [00:29<00:00, 1.81it/s] Total progress: 100%|██████████████████████████████████████████████████████████████████| 53/53 [00:29<00:00, 1.84s/it] ``` ### Additional information This has been an issue since Hires fix came out. Been fighting this the entire time since November when started the hobby. What is infuriating is the preview shows the image being rendered correctly until a certain point, which I assume is in the Hires. fix at which it iterates into the cloned faces. Fewer steps lessons the effect, but never solve it. Even at None and 0.1, the image is strongly effected.

JoyBell2 commented 1 year ago

I posted the same issue on Reddit with more examples and comments. https://www.reddit.com/r/StableDiffusion/comments/15apxys/hires_fix_causes_clones_at_all_settings_please/

ClashSAN commented 1 year ago

There is a couple of face in painting extensions that should fit the ideal. webui hires fix = run a text2img -> switch and do img2img midway.

There is no smart targeting system in stable diffusion architecture.

JoyBell2 commented 1 year ago

Doesn't explain to me why the text2img does the correct thing but img2img can't. For me, this breaks all prompts with more than one character.

leucome commented 1 year ago

Technically speaking it is not a bug. As there is nothing in higres fix that control multiple concept. After the upscale it try to paint something that look alike on top. Most of the time the woman likeness is taking over.

Some people fix this issue by using really low denoising like 0.15. With upscaller like R-ESRGAN 4X or LDSR the result can be ok butt it is not that great. It is missing details and quite blurry.

It is possible to make a prompt in a way to give more priority the the concept that disappear. By example like this.

photograph of (Ryan Reynolds) dancing with Anne Hathaway at a Brazilian night club, well dressed up for a night out, by Vogue

It will work sometime but it is still really likely to fail. You can also go in setting user interface then check "Hires fix: show hires prompt and negative prompt " . This way you can add a custom prompt for the hires fix. This really increase the chance of success.

Alternative option... This is an hires fix example made with Restart sampler currently in the Pull Request #11850. This sampler is particularly good at keeping concept separation. It only made Ryan slightly feminine and Anne slightly masculine. (note: I also used earlier prompt suggestion) 00010-1742889144

Still to guarantee proper upscaling of multiple concept it is usually required to use img2img and controlnet. Tile and canny are good controlnet model to do that. The option I found that give really good result is Tilled Diffusion extension with noise inversion combined with controlnet canny an tile all at once. This solution is a bit complicated to setup for the first time. But once you nail down the setting the result are worth it.

This example is made with Tilled Diffusion, 20 step noise inversion and controlnet canny+tile at 0.6 control Weight. It look really similar to the base image I used. 00024-1742889144

Here the base image before the hires fix/upscale. base-00011-1742889144

There is somebody who asked for hires fix that can use base generation as input for controlnet instead of a picture... So far nobody implemented this. So we still need to go to image to image or at least generate a base image first then put it back in control net and re-generate with hires.

MrNeon commented 1 year ago

As others have stated the issue is that concepts leak as you get further away from the base resolution the model was trained at. Adding to what @leucome said I've been having great fun using regional prompter to keep things separated. It works really great in attention mode although lora/hypernetwork suffers because they are applied to the whole image. https://github.com/hako-mikan/sd-webui-regional-prompter

AUTOMATIC1111 / stable-diffusion-webui