roseyyy2022ai
commented
1 year ago Is there an existing issue for this?
- [x] I have searched the existing issues and checked the recent builds/commits
What happened?
I launch a batch render in Img2Img with Control net.
I'm using the Tile Control net and Upscale Script.
I launch a batch render and everything is fine. Then after a random number of ok renders, it hangs. Every time is a different number of renders completed successfully.
The problem seems to gradually get worse (Less images completed before the process hangs) the more times you re-launch the batch.
Steps to reproduce the problem
Setup a batch render in Img2Img using control net, tile, and upscale script.
What should have happened?
Every image in the batch directory should render completely. But instead the process hangs somewhere between 20 and 2 images completed.
Commit where the problem happens
python: 3.10.9 • torch: 1.13.1+cu117 • xformers: 0.0.16rc425 • gradio: 3.23.0 • commit: • checkpoint: 44f90a0972
What platforms do you use to access the UI ?
Windows
What browsers do you use to access the UI ?
Google Chrome
Command Line Arguments
NoList of extensions
sd-webui-additional-networks 75141070 (Wed May 3 09:52:01 2023) sd-webui-controlnet 5d387abf (Thu May 4 02:03:32 2023) ultimate-upscale-for-automatic1111 756bb505 (Fri May 5 00:22:21 2023)
Console logs
venv "I:\AI\Auto1111\stable-diffusion-webui\venv\Scripts\Python.exe" Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] Commit hash: <none> Installing requirements for Web UI Launching Web UI with arguments: --no-half --precision full --force-enable-xformers --xformers --port 7862 [AddNet] Updating model hashes... 0it [00:00, ?it/s] [AddNet] Updating model hashes... 0it [00:00, ?it/s] ControlNet v1.1.134 ControlNet v1.1.134 Loading weights [44f90a0972] from I:\AI\Auto1111\stable-diffusion-webui\models\Stable-diffusion\protogenX34Photoreal_1.safetensors Creating model from config: I:\AI\Auto1111\stable-diffusion-webui\configs\v1-inference.yaml LatentDiffusion: Running in eps-prediction mode DiffusionWrapper has 859.52 M params. Applying xformers cross attention optimization. Textual inversion embeddings loaded(83): Textual inversion embeddings skipped(20): Model loaded in 6.4s (load weights from disk: 0.6s, create model: 0.5s, apply weights to model: 1.8s, move model to device: 1.9s, load textual inversion embeddings: 1.5s). Running on local URL: http://127.0.0.1:7862 To create a public link, set `share=True` in `launch()`. Startup time: 13.3s (import torch: 1.4s, import gradio: 1.0s, import ldm: 0.7s, other imports: 1.0s, load scripts: 1.2s, load SD checkpoint: 7.0s, create ui: 0.8s, gradio launch: 0.1s). controlnet batch mode Will process 107 images, creating 1 new images for each. Canva size: 3072x3072 Image size: 1536x1536 Scale factor: 2 Upscaling iteration 1 with scale factor 2 Tile size: 768x768 Tiles amount: 16 Grid: 4x4 Redraw enabled: True Seams fix mode: NONE Loading model: control_v11f1e_sd15_tile [a371b31b] Loaded state_dict from [I:\AI\Auto1111\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.pth] Loading config: I:\AI\Auto1111\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.yaml ControlNet model control_v11f1e_sd15_tile [a371b31b] loaded. Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 76/1216 [00:34<08:50, 2.15it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:35<00:00, 2.13it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 152/1216 [01:13<08:23, 2.11it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.11it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 228/1216 [01:53<07:51, 2.10it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.10it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 304/1216 [02:33<07:15, 2.09it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.09it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 380/1216 [03:12<06:40, 2.09it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.09it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 456/1216 [03:52<06:04, 2.09it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.09it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█▌ | 532/1216 [04:32<05:29, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.09it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████▌ | 608/1216 [05:12<04:52, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████▍ | 684/1216 [05:52<04:16, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████████████████▍ | 760/1216 [06:32<03:40, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████████████████▎ | 836/1216 [07:12<03:03, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.07it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████████████████████████████▎ | 912/1216 [07:52<02:26, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████████████████████████████▏ | 988/1216 [08:32<01:49, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]██████████████████████████████████████████▎ | 1064/1216 [09:13<01:13, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.07it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]████████████████████████████████████████████████▏ | 1140/1216 [09:53<00:36, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.07it/s] Total progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 1216/1216 [10:40<00:00, 1.90it/s] Canva size: 3072x3072██████████████████████████████████████████████████████████████████████████████████████████████| 1216/1216 [10:40<00:00, 2.07it/s] Image size: 1536x1536 Scale factor: 2 Upscaling iteration 1 with scale factor 2 Tile size: 768x768 Tiles amount: 16 Grid: 4x4 Redraw enabled: True Seams fix mode: NONE Loading model from cache: control_v11f1e_sd15_tile [a371b31b] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 76/1216 [00:36<09:09, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 152/1216 [01:16<08:33, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.07it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 228/1216 [01:56<07:57, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.07it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 304/1216 [02:36<07:21, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.07it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 380/1216 [03:16<06:44, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.07it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 456/1216 [03:56<06:07, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.07it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█▌ | 532/1216 [04:37<05:30, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████▌ | 608/1216 [05:17<04:53, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████▍ | 684/1216 [05:57<04:16, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████████████████▍ | 760/1216 [06:38<03:39, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████████████████▎ | 836/1216 [07:18<03:03, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]███████████████████████████████▎ | 912/1216 [07:58<02:26, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]█████████████████████████████████████▏ | 988/1216 [08:38<01:49, 2.07it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]██████████████████████████████████████████▎ | 1064/1216 [09:18<01:13, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b]████████████████████████████████████████████████▏ | 1140/1216 [09:58<00:36, 2.08it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.08it/s] Total progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 1216/1216 [10:45<00:00, 1.88it/s] Canva size: 3072x3072██████████████████████████████████████████████████████████████████████████████████████████████| 1216/1216 [10:45<00:00, 2.08it/s] Image size: 1536x1536 Scale factor: 2 Upscaling iteration 1 with scale factor 2 Tile size: 768x768 Tiles amount: 16 Grid: 4x4 Redraw enabled: True Seams fix mode: NONE Loading model from cache: control_v11f1e_sd15_tile [a371b31b] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:36<00:00, 2.09it/s] Loading model from cache: control_v11f1e_sd15_tile [a371b31b] | 76/1216 [00:35<09:06, 2.09it/s] Loading preprocessor: tile_resample Pixel Perfect Mode Enabled. resize_mode = ResizeMode.INNER_FIT raw_H = 832 raw_W = 832 target_H = 832 target_W = 832 estimation = 832.0 preprocessor resolution = 832 0%| | 0/76 [00:00<?, ?it/s] Total progress: 6%|██████ | 76/1216 [00:46<11:34, 1.64it/s] controlnet batch mode██████Additional information
The problem seems to gradually worsen, so maybe memory? As in each time I re-run the batch, without restarting the session, less and less images are rendered correctly before the hang.
The hang is acting more like the process is complete, it's not a crash. So I can restart the process again from the same session.
I also noticed that if the source directory has a Thumbs.db file that can cause this same batch rendering problems. I just fixed that by blowing away the thumbs file.
Thank God I'm not the only experiencing this!!! lmk when there's a fix guys
Marina-chu
E2GO
weiweizwc98
Is there an existing issue for this?
What happened?
I launch a batch render in Img2Img with Control net.
I'm using the Tile Control net and Upscale Script.
I launch a batch render and everything is fine. Then after a random number of ok renders, it hangs. Every time is a different number of renders completed successfully.
The problem seems to gradually get worse (Less images completed before the process hangs) the more times you re-launch the batch.
Steps to reproduce the problem
Setup a batch render in Img2Img using control net, tile, and upscale script.
What should have happened?
Every image in the batch directory should render completely. But instead the process hangs somewhere between 20 and 2 images completed.
Commit where the problem happens
python: 3.10.9 • torch: 1.13.1+cu117 • xformers: 0.0.16rc425 • gradio: 3.23.0 • commit: • checkpoint: 44f90a0972
What platforms do you use to access the UI ?
Windows
What browsers do you use to access the UI ?
Google Chrome
Command Line Arguments
List of extensions
sd-webui-additional-networks 75141070 (Wed May 3 09:52:01 2023)
sd-webui-controlnet 5d387abf (Thu May 4 02:03:32 2023)
ultimate-upscale-for-automatic1111 756bb505 (Fri May 5 00:22:21 2023)
Console logs
Additional information
The problem seems to gradually worsen, so maybe memory? As in each time I re-run the batch, without restarting the session, less and less images are rendered correctly before the hang.
The hang is acting more like the process is complete, it's not a crash. So I can restart the process again from the same session.
I also noticed that if the source directory has a Thumbs.db file that can cause this same batch rendering problems. I just fixed that by blowing away the thumbs file.