search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.43k stars 394 forks source link

[Issue]: 2-4 images go black. Failed to validate samples (Intel Arc A770) #2901

Closed AndrewRainfall closed 6 months ago

AndrewRainfall commented 6 months ago

Issue Description

2-4 images go black with this error, in most cases only one image out of four is actually generated:

20:00:38-781232 ERROR Failed to validate samples: sample=(512, 512, 3) invalid=786432 20:00:38-787233 WARNING Attempted to correct samples: min=0.00 max=0.00 mean=0.00

No nan errors and any combination of those settings didn't help: Precision type Autocast, Full; Device precision type FP32. FP16, BF16; Full precision for model (--no-half), Full precision for VAE (--no-half-vae), Upcast sampling.

I tried with different models & VAEs (but only 1.5).

I'm also concerned about "Error running git" messages & "Cmd('git') failed due to: exit code(128)", but I don't know are those connected with the issue or not.

Version Platform Description

New build downloaded yesterday, Win10, Google Chrome, Intel GPU Arc A770 16G VRAM with the latest drivers but without ReBAR

19:56:25-839716 INFO Starting SD.Next 19:56:25-845717 INFO Logger: file="E:\automatic\sdnext.log" level=INFO size=524555 mode=append 19:56:25-848717 INFO Python 3.10.6 on Windows 19:56:26-236785 INFO Version: app=sd.next updated=2024-02-22 hash=d6906c1b url=https://github.com/vladmandic/automatic/tree/master 19:56:27-827456 INFO Latest published version: c5f6c25a37858daedc6c59988d2e684c4fa7ddfb 2024-02-23T14:53:07Z 19:56:27-839459 INFO Platform: arch=AMD64 cpu=AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD system=Windows release=Windows-10-10.0.19045-SP0 python=3.10.6 19:56:27-844960 INFO Intel OneAPI Toolkit detected

Relevant log output

Using VENV: E:\automatic\venv
19:56:25-839716 INFO     Starting SD.Next
19:56:25-845717 INFO     Logger: file="E:\automatic\sdnext.log" level=INFO size=524555 mode=append
19:56:25-848717 INFO     Python 3.10.6 on Windows
19:56:26-236785 INFO     Version: app=sd.next updated=2024-02-22 hash=d6906c1b
                         url=https://github.com/vladmandic/automatic/tree/master
19:56:27-827456 INFO     Latest published version: c5f6c25a37858daedc6c59988d2e684c4fa7ddfb 2024-02-23T14:53:07Z
19:56:27-839459 INFO     Platform: arch=AMD64 cpu=AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD system=Windows
                         release=Windows-10-10.0.19045-SP0 python=3.10.6
19:56:27-844960 INFO     Intel OneAPI Toolkit detected
19:56:28-111099 WARNING  Deleted files: ['repositories/codeformer/basicsr/VERSION',
                         'repositories/codeformer/basicsr/__init__.py',
                         'repositories/codeformer/basicsr/archs/__init__.py',
                         'repositories/codeformer/basicsr/archs/arcface_arch.py',
                         'repositories/codeformer/basicsr/archs/arch_util.py',
                         'repositories/codeformer/basicsr/archs/codeformer_arch.py',
                         'repositories/codeformer/basicsr/archs/rrdbnet_arch.py',
                         'repositories/codeformer/basicsr/archs/vgg_arch.py',
                         'repositories/codeformer/basicsr/archs/vqgan_arch.py',
                         'repositories/codeformer/basicsr/data/__init__.py',
                         'repositories/codeformer/basicsr/data/data_sampler.py',
                         'repositories/codeformer/basicsr/data/data_util.py',
                         'repositories/codeformer/basicsr/data/prefetch_dataloader.py',
                         'repositories/codeformer/basicsr/data/transforms.py',
                         'repositories/codeformer/basicsr/losses/__init__.py',
                         'repositories/codeformer/basicsr/losses/loss_util.py',
                         'repositories/codeformer/basicsr/losses/losses.py',
                         'repositories/codeformer/basicsr/metrics/__init__.py',
                         'repositories/codeformer/basicsr/metrics/metric_util.py',
                         'repositories/codeformer/basicsr/metrics/psnr_ssim.py',
                         'repositories/codeformer/basicsr/models/__init__.py',
                         'repositories/codeformer/basicsr/ops/__init__.py',
                         'repositories/codeformer/basicsr/ops/dcn/__init__.py',
                         'repositories/codeformer/basicsr/ops/dcn/deform_conv.py',
                         'repositories/codeformer/basicsr/ops/dcn/src/deform_conv_cuda.cpp',
                         'repositories/codeformer/basicsr/ops/dcn/src/deform_conv_cuda_kernel.cu',
                         'repositories/codeformer/basicsr/ops/dcn/src/deform_conv_ext.cpp',
                         'repositories/codeformer/basicsr/ops/fused_act/__init__.py',
                         'repositories/codeformer/basicsr/ops/fused_act/fused_act.py',
                         'repositories/codeformer/basicsr/ops/fused_act/src/fused_bias_act.cpp',
                         'repositories/codeformer/basicsr/ops/fused_act/src/fused_bias_act_kernel.cu',
                         'repositories/codeformer/basicsr/ops/upfirdn2d/__init__.py',
                         'repositories/codeformer/basicsr/ops/upfirdn2d/src/upfirdn2d.cpp',
                         'repositories/codeformer/basicsr/ops/upfirdn2d/src/upfirdn2d_kernel.cu',
                         'repositories/codeformer/basicsr/ops/upfirdn2d/upfirdn2d.py',
                         'repositories/codeformer/basicsr/setup.py', 'repositories/codeformer/basicsr/train.py',
                         'repositories/codeformer/basicsr/utils/__init__.py',
                         'repositories/codeformer/basicsr/utils/dist_util.py',
                         'repositories/codeformer/basicsr/utils/download_util.py',
                         'repositories/codeformer/basicsr/utils/file_client.py',
                         'repositories/codeformer/basicsr/utils/img_util.py',
                         'repositories/codeformer/basicsr/utils/lmdb_util.py',
                         'repositories/codeformer/basicsr/utils/logger.py',
                         'repositories/codeformer/basicsr/utils/matlab_functions.py',
                         'repositories/codeformer/basicsr/utils/misc.py',
                         'repositories/codeformer/basicsr/utils/options.py',
                         'repositories/codeformer/basicsr/utils/realesrgan_utils.py',
                         'repositories/codeformer/basicsr/utils/registry.py',
                         'repositories/codeformer/basicsr/utils/video_util.py',
                         'repositories/codeformer/Jinja2-3.1.3.dist-info/',
                         'repositories/codeformer/Markdown-3.5.2.dist-info/',
                         'repositories/codeformer/MarkupSafe-2.1.5.dist-info/', 'repositories/codeformer/absl/',
                         'repositories/codeformer/absl_py-2.1.0.dist-info/',
                         'repositories/codeformer/addict-2.4.0.dist-info/', 'repositories/codeformer/addict/',
                         'repositories/codeformer/basicsr-1.4.2.dist-info/',
                         'repositories/codeformer/basicsr/basicsr/',
                         'repositories/codeformer/certifi-2024.2.2.dist-info/', 'repositories/codeformer/certifi/',
                         'repositories/codeformer/charset_normalizer-3.3.2.dist-info/',
                         'repositories/codeformer/charset_normalizer/',
                         'repositories/codeformer/colorama-0.4.6.dist-info/', 'repositories/codeformer/colorama/',
                         'repositories/codeformer/cv2/', 'repositories/codeformer/filelock-3.13.1.dist-info/',
                         'repositories/codeformer/filelock/', 'repositories/codeformer/fsspec-2024.2.0.dist-info/',
                         'repositories/codeformer/fsspec/', 'repositories/codeformer/functorch/',
                         'repositories/codeformer/future-1.0.0.dist-info/', 'repositories/codeformer/future/',
                         'repositories/codeformer/google/', 'repositories/codeformer/grpc/',
                         'repositories/codeformer/grpcio-1.62.0.dist-info/',
                         'repositories/codeformer/idna-3.6.dist-info/', 'repositories/codeformer/idna/',
                         'repositories/codeformer/imageio-2.34.0.dist-info/', 'repositories/codeformer/imageio/',
                         'repositories/codeformer/importlib_metadata-7.0.1.dist-info/',
                         'repositories/codeformer/importlib_metadata/', 'repositories/codeformer/isympy.py',
                         'repositories/codeformer/jinja2/', 'repositories/codeformer/lazy_loader-0.3.dist-info/',
                         'repositories/codeformer/lazy_loader/', 'repositories/codeformer/libfuturize/',
                         'repositories/codeformer/libpasteurize/', 'repositories/codeformer/lmdb-1.4.1.dist-info/',
                         'repositories/codeformer/lmdb/', 'repositories/codeformer/markdown/',
                         'repositories/codeformer/markupsafe/', 'repositories/codeformer/mpmath-1.3.0.dist-info/',
                         'repositories/codeformer/mpmath/', 'repositories/codeformer/networkx-3.2.1.dist-info/',
                         'repositories/codeformer/networkx/', 'repositories/codeformer/numpy/']
19:56:28-196615 WARNING  Modified files: ['repositories/codeformer/isympy.py']
19:56:28-301633 INFO     Startup: standard
19:56:28-303635 INFO     Verifying requirements
19:56:28-322137 INFO     Verifying packages
19:56:28-325137 INFO     Verifying submodules
19:56:35-210392 ERROR    Error running git: extensions-builtin/sd-extension-chainner / branch
19:56:35-301408 ERROR    Error running git: extensions-builtin/sd-extension-system-info / branch
19:56:35-391924 ERROR    Error running git: extensions-builtin/sd-webui-agent-scheduler / branch
19:56:35-485440 ERROR    Error running git: extensions-builtin/sd-webui-controlnet / branch
19:56:35-575455 ERROR    Error running git: extensions-builtin/stable-diffusion-webui-images-browser / branch
19:56:35-663970 ERROR    Error running git: extensions-builtin/stable-diffusion-webui-rembg / branch
19:56:35-754486 ERROR    Error running git: modules/k-diffusion / branch
19:56:35-854003 ERROR    Error running git: wiki / branch
19:56:40-582827 INFO     Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg']
19:56:40-585328 INFO     Verifying requirements
19:56:40-603331 WARNING  Setup complete with errors: 8
19:56:40-605332 WARNING  See log file for more details: E:\automatic\sdnext.log
19:56:40-702848 INFO     Extension preload: {'extensions-builtin': 0.09, 'extensions': 0.0}
19:56:40-705849 INFO     Command line args: ['--use-ipex', '--autolaunch'] autolaunch=True use_ipex=True
19:56:49-394916 INFO     Load packages: {'torch': '2.1.0a0+cxx11.abi', 'diffusers': '0.26.3', 'gradio': '3.43.2'}
19:56:50-633986 INFO     Engine: backend=Backend.DIFFUSERS compute=ipex device=xpu attention="Scaled-Dot-Product" mode=no_grad
19:56:50-637987 INFO     Device: device=Intel(R) Arc(TM) A770 Graphics n=1 ipex=2.1.10+xpu
19:56:51-867224 INFO     Available VAEs: path="E:\models\VAE" items=3
19:56:51-870224 INFO     Disabled extensions: ['sd-webui-controlnet']
19:56:52-233458 INFO     Available models: path="E:\models\Stable-diffusion" items=22 time=0.36
19:56:52-239459 ERROR    GFPGan failed to initialize: No module named 'gfpgan'
19:56:52-471632 INFO     Extension: script='extensions-builtin\Lora\scripts\lora_script.py' 19:56:52-378669 INFO     LoRA networks: available=17 folders=2
19:56:53-082238 INFO     Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py' Using sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3
19:56:53-482586 INFO     Extensions init time: 1.24 animatediff.py=0.10 Lora=0.11 sd-extension-chainner=0.13 sd-webui-agent-scheduler=0.47 stable-diffusion-webui-images-browser=0.38
19:56:53-499089 ERROR    Error loading upscaler: realesrgan No module named 'basicsr.archs'
19:56:53-534596 INFO     UI theme: name="black-teal" style=Dark base=sdnext.css
19:56:55-115871 INFO     Extension list is empty: refresh required
19:56:55-166881 ERROR    Extension: failed reading data from git repo=sd-extension-chainner: Cmd('git') failed due to: exit code(128)
                           cmdline: git remote get-url --all origin
                           stderr: 'fatal: detected dubious ownership in repository at 'E:/automatic/extensions-builtin/sd-extension-chainner'
                         'E:/automatic/extensions-builtin/sd-extension-chainner/.git' is on a file system that does not record ownership
                         To add an exception for this directory, call:

                                 git config --global --add safe.directory E:/automatic/extensions-builtin/sd-extension-chainner'
19:56:55-227890 ERROR    Extension: failed reading data from git repo=sd-extension-system-info: Cmd('git') failed due to: exit code(128)
                           cmdline: git remote get-url --all origin
                           stderr: 'fatal: detected dubious ownership in repository at 'E:/automatic/extensions-builtin/sd-extension-system-info'
                         'E:/automatic/extensions-builtin/sd-extension-system-info/.git' is on a file system that does not record ownership
                         To add an exception for this directory, call:

                                 git config --global --add safe.directory E:/automatic/extensions-builtin/sd-extension-system-info'
19:56:55-282900 ERROR    Extension: failed reading data from git repo=sd-webui-agent-scheduler: Cmd('git') failed due to: exit code(128)
                           cmdline: git remote get-url --all origin
                           stderr: 'fatal: detected dubious ownership in repository at 'E:/automatic/extensions-builtin/sd-webui-agent-scheduler'
                         'E:/automatic/extensions-builtin/sd-webui-agent-scheduler/.git' is on a file system that does not record ownership
                         To add an exception for this directory, call:

                                 git config --global --add safe.directory E:/automatic/extensions-builtin/sd-webui-agent-scheduler'
19:56:55-336409 ERROR    Extension: failed reading data from git repo=sd-webui-controlnet: Cmd('git') failed due to: exit code(128)
                           cmdline: git remote get-url --all origin
                           stderr: 'fatal: detected dubious ownership in repository at 'E:/automatic/extensions-builtin/sd-webui-controlnet'
                         'E:/automatic/extensions-builtin/sd-webui-controlnet/.git' is on a file system that does not record ownership
                         To add an exception for this directory, call:

                                 git config --global --add safe.directory E:/automatic/extensions-builtin/sd-webui-controlnet'
19:56:55-393420 ERROR    Extension: failed reading data from git repo=stable-diffusion-webui-images-browser: Cmd('git') failed due to: exit code(128)
                           cmdline: git remote get-url --all origin
                           stderr: 'fatal: detected dubious ownership in repository at 'E:/automatic/extensions-builtin/stable-diffusion-webui-images-browser'
                         'E:/automatic/extensions-builtin/stable-diffusion-webui-images-browser/.git' is on a file system that does not record ownership
                         To add an exception for this directory, call:

                                 git config --global --add safe.directory E:/automatic/extensions-builtin/stable-diffusion-webui-images-browser'
19:56:55-447429 ERROR    Extension: failed reading data from git repo=stable-diffusion-webui-rembg: Cmd('git') failed due to: exit code(128)
                           cmdline: git remote get-url --all origin
                           stderr: 'fatal: detected dubious ownership in repository at 'E:/automatic/extensions-builtin/stable-diffusion-webui-rembg'
                         'E:/automatic/extensions-builtin/stable-diffusion-webui-rembg/.git' is on a file system that does not record ownership
                         To add an exception for this directory, call:

                                 git config --global --add safe.directory E:/automatic/extensions-builtin/stable-diffusion-webui-rembg'
19:56:55-942056 INFO     Local URL: http://127.0.0.1:7860/
19:56:55-944557 INFO     Initializing middleware
19:56:56-197100 INFO     [AgentScheduler] Task queue is empty
19:56:56-199600 INFO     [AgentScheduler] Registering APIs
19:56:56-582668 INFO     Select: model="cyberrealistic_v41BackToBasics [41b6846108]"
19:56:56-956162 INFO     Setting Torch parameters: device=xpu dtype=torch.bfloat16 vae=torch.bfloat16 unet=torch.bfloat16 context=no_grad fp16=True bf16=True optimization=Scaled-Dot-Product
19:56:56-960663 INFO     Loading VAE: model=E:\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors source=settings
19:56:56-964163 INFO     Autodetect: vae="Stable Diffusion" class=StableDiffusionPipeline file="E:\models\Stable-diffusion\cyberrealistic_v41BackToBasics.safetensors" size=2034MB
19:56:57-262715 INFO     Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="E:\models\Stable-diffusion\cyberrealistic_v41BackToBasics.safetensors" size=2034MB
19:57:04-692716 INFO     IPEX Optimize: time=1.30
19:57:05-398173 INFO     Load embeddings: loaded=7 skipped=1 time=0.69
19:57:05-615211 INFO     Load model: time=8.81 load=8.81 native=512 {'ram': {'used': 1.73, 'total': 31.95}, 'gpu': {'used': 2.13, 'total': 15.54}, 'retries': 0, 'oom': 0}
19:57:05-621712 INFO     Startup time: 24.91 torch=7.26 olive=0.10 gradio=1.33 libraries=2.42 extensions=1.24 models=0.36 ui-en=0.30 ui-txt2img=0.07 ui-img2img=0.08 ui-control=0.12 ui-settings=0.28 ui-extensions=0.94 ui-defaults=0.10
                         launch=0.37 api=0.14 app-started=0.50 checkpoint=9.04
19:57:05-634714 INFO     Launching browser
19:57:10-556572 INFO     MOTD: N/A
19:57:12-970493 INFO     Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36
Progress  1.17it/s █████████████████████████████████ 100% 25/25 00:21 00:00 Base
20:00:38-168941 ERROR    Failed to validate samples: sample=(512, 512, 3) invalid=786432
20:00:38-176943 WARNING  Attempted to correct samples: min=0.00 max=0.00 mean=0.00
20:00:38-279461 INFO     Saving: image="outputs\text\00120-cyberrealistic_v41BackToBasics-masterpiece best quality 1 2 BREAK hatsune miku.jpg" type=JPEG resolution=512x512 size=0
20:00:38-382479 ERROR    Failed to validate samples: sample=(512, 512, 3) invalid=786432
20:00:38-388481 WARNING  Attempted to correct samples: min=0.00 max=0.00 mean=0.00
20:00:38-489998 INFO     Saving: image="outputs\text\00121-cyberrealistic_v41BackToBasics-masterpiece best quality 1 2 BREAK hatsune miku.jpg" type=JPEG resolution=512x512 size=0
20:00:38-680714 INFO     Saving: image="outputs\text\00122-cyberrealistic_v41BackToBasics-masterpiece best quality 1 2 BREAK hatsune miku.jpg" type=JPEG resolution=512x512 size=0
20:00:38-781232 ERROR    Failed to validate samples: sample=(512, 512, 3) invalid=786432
20:00:38-787233 WARNING  Attempted to correct samples: min=0.00 max=0.00 mean=0.00
20:00:38-886982 INFO     Saving: image="outputs\text\00123-cyberrealistic_v41BackToBasics-masterpiece best quality 1 2 BREAK hatsune miku.jpg" type=JPEG resolution=512x512 size=0
20:00:38-981999 INFO     Processed: images=4 time=31.96 its=3.13 memory={'ram': {'used': 2.04, 'total': 31.95}, 'gpu': {'used': 3.3, 'total': 15.54}, 'retries': 0, 'oom': 0}
20:00:39-082699 INFO     Saving: image="outputs\grids\00028-cyberrealistic_v41BackToBasics-masterpiece best quality 1 2 BREAK hatsune miku-grid.jpg" type=JPEG resolution=1024x1024 size=0

Backend

Diffusers

Branch

Master

Model

SD 1.5

Acknowledgements

brknsoul commented 6 months ago

You can fix dubious ownership by git config --global --add safe.directory *

Try running Precision: Full, FP32, and the two tick boxes underneath (the ones with --no-half and --no-half-vae in parentheses) and see if that solves black images.

AndrewRainfall commented 6 months ago

Thanks for the dubious ownership fix, I tried smth like this before but it didn't work without a star.

The second part didn't help and I already tried it before. Is it just me or does this improve the image quality a bit?

Upd.: Tried with SDXL (juggernautXL). Same issue, but even worse - bigger chance to get all images black and even if one of the images is not black it is corrupted most of the time (if not always).

With compute settings it's still probably the same: at least one image turned black before I stopped generation (cuz generation time increased from 1 minute to 1 hour)

Disty0 commented 6 months ago

You have way too many deleted files, modified files and permission issues. Try with a new install and without modifying anything.

AndrewRainfall commented 6 months ago

You have way too many deleted files, modified files and permission issues. Try with a new install and without modifying anything.

They happened after I tried to fix the GFPGAN & basicsr errors. Black images was from fresh installation. I'll revert all & post the results.

AndrewRainfall commented 6 months ago

Deleted the automatic folder & git cloned automatic to another internal ssd (before it was on external ssd). Does installing on internal drive give better performance?

For some reason I got way less errors during launch, but it didn't resolve the issue.

Using VENV: C:\automatic\venv 13:37:25-082523 INFO Starting SD.Next 13:37:25-089025 INFO Logger: file="C:\automatic\sdnext.log" level=INFO size=61345 mode=append 13:37:25-093026 INFO Python 3.10.6 on Windows 13:37:29-378779 INFO Version: app=sd.next updated=2024-02-23 hash=c5f6c25a url=https://github.com/vladmandic/automatic/tree/master 13:37:30-721015 INFO Platform: arch=AMD64 cpu=AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD system=Windows release=Windows-10-10.0.19045-SP0 python=3.10.6 13:37:30-727016 INFO Intel OneAPI Toolkit detected 13:37:32-222779 INFO Extensions: disabled=[] 13:37:32-225280 INFO Extensions: enabled=['Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg'] extensions-builtin 13:37:32-236281 INFO Extensions: enabled=[] extensions 13:37:32-238782 INFO Startup: quick launch 13:37:32-240283 INFO Verifying requirements 13:37:32-257786 INFO Verifying packages 13:37:32-261786 INFO Extensions: disabled=[] 13:37:32-264287 INFO Extensions: enabled=['Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg'] extensions-builtin 13:37:32-274789 INFO Extensions: enabled=[] extensions 13:37:32-285791 INFO Command line args: ['--use-ipex', '--autolaunch'] autolaunch=True use_ipex=True 13:37:42-572599 INFO Load packages: {'torch': '2.1.0a0+cxx11.abi', 'diffusers': '0.26.3', 'gradio': '3.43.2'} 13:37:43-789813 INFO Engine: backend=Backend.DIFFUSERS compute=ipex device=xpu attention="Scaled-Dot-Product" mode=no_grad 13:37:43-794314 INFO Device: device=Intel(R) Arc(TM) A770 Graphics n=1 ipex=2.1.10+xpu 13:37:45-594552 INFO Available VAEs: path="E:\models\VAE" items=3 13:37:45-598552 INFO Disabled extensions: ['sd-webui-controlnet'] 13:37:46-491346 INFO Available models: path="E:\models\Stable-diffusion" items=22 time=0.89 13:37:46-856910 INFO Extension: script='extensions-builtin\Lora\scripts\lora_script.py' 13:37:46-851909 INFO LoRA networks: available=17 folders=2 13:37:47-410508 INFO Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py' Using sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3 13:37:47-798575 INFO UI theme: name="black-teal" style=Dark base=sdnext.css 13:37:51-694761 INFO Extension list is empty: refresh required 13:37:53-706114 INFO Local URL: http://127.0.0.1:7860/ 13:37:54-177698 INFO [AgentScheduler] Task queue is empty 13:37:54-180199 INFO [AgentScheduler] Registering APIs 13:37:54-365731 INFO Select: model="cyberrealistic_v41BackToBasics [41b6846108]" 13:37:54-423240 INFO Torch override dtype: no-half set 13:37:54-425241 INFO Torch override VAE dtype: no-half set 13:37:54-427242 INFO Setting Torch parameters: device=xpu dtype=torch.float32 vae=torch.float32 unet=torch.float32 context=no_grad fp16=None bf16=None optimization=Scaled-Dot-Product 13:37:54-516756 INFO Loading VAE: model=E:\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors source=settings 13:37:54-608272 INFO Autodetect: vae="Stable Diffusion" class=StableDiffusionPipeline file="E:\models\Stable-diffusion\cyberrealistic_v41BackToBasics.safetensors" size=2034MB 13:37:54-972337 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="E:\models\Stable-diffusion\cyberrealistic_v41BackToBasics.safetensors" size=2034MB 13:38:00-847825 INFO IPEX Optimize: time=0.08 13:38:00-942342 INFO Load embeddings: loaded=0 skipped=0 time=0.00 13:38:01-163880 INFO Load model: time=6.58 config=0.05 load=6.52 native=512 {'ram': {'used': 2.03, 'total': 31.95}, 'gpu': {'used': 4.09, 'total': 15.54}, 'retries': 0, 'oom': 0} 13:38:01-171882 INFO Startup time: 28.88 torch=8.57 olive=0.11 gradio=1.60 libraries=2.52 samplers=0.15 vae=0.33 extensions=1.15 models=0.89 face-restore=0.10 ui-en=2.36 ui-txt2img=0.07 ui-img2img=0.08 ui-control=0.13 ui-settings=0.47 ui-extensions=2.32 ui-defaults=0.11 launch=0.20 api=0.14 app-started=0.51 checkpoint=6.81 13:38:01-185884 INFO Launching browser 13:38:07-942707 INFO MOTD: N/A 13:38:11-304799 INFO Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Progress 1.51s/it █████████████████████████████████ 100% 25/25 00:37 00:00 Base 13:39:47-448609 INFO Saving: image="outputs\text\00008-cyberrealistic_v41BackToBasics-hatsune miku in the forest daylight upper body.jpg" type=JPEG resolution=512x512 size=0 13:39:47-466612 INFO Saving: image="outputs\text\00009-cyberrealistic_v41BackToBasics-hatsune miku in the forest daylight upper body.jpg" type=JPEG resolution=512x512 size=0 13:39:47-478114 ERROR Failed to validate samples: sample=(512, 512, 3) invalid=786432 13:39:47-484615 WARNING Attempted to correct samples: min=0.00 max=0.00 mean=0.00 13:39:47-493616 INFO Saving: image="outputs\text\00010-cyberrealistic_v41BackToBasics-hatsune miku in the forest daylight upper body.jpg" type=JPEG resolution=512x512 size=0 13:39:47-501118 ERROR Failed to validate samples: sample=(512, 512, 3) invalid=786432 13:39:47-507119 WARNING Attempted to correct samples: min=0.00 max=0.00 mean=0.00 13:39:47-516121 INFO Saving: image="outputs\text\00011-cyberrealistic_v41BackToBasics-hatsune miku in the forest daylight upper body.jpg" type=JPEG resolution=512x512 size=0 13:39:47-522122 INFO Processed: images=4 time=52.41 its=1.91 memory={'ram': {'used': 2.6, 'total': 31.95}, 'gpu': {'used': 5.88, 'total': 15.54}, 'retries': 0, 'oom': 0}

Tried again to turn on recomended compute settings, but they didn't help either.

Disty0 commented 6 months ago

Do you have any ipex specific environment variable set? I am suspecting forced 4GB workaround for ARC doesn't get applied here. And if this is the cause, FP32 / no-half will make thing worse for you.

You can try the Dynamic Attention SDP with slicing rate 4 or less from settings. Or try setting IPEX_SDPA_SLICE_TRIGGER_RATE and IPEX_ATTENTION_SLICE_RATE environment variables to 4 or less from cmd.

And try disabling IPEX Optimize from settings. It might be bad on Windows.

AndrewRainfall commented 6 months ago

Do you have any ipex specific environment variable set? = I didn't installed anything with ipex aside SD.Next, so I guess no.

You can try the Dynamic Attention SDP with slicing rate 4 or less from settings. Or try setting IPEX_SDPA_SLICE_TRIGGER_RATE and IPEX_ATTENTION_SLICE_RATE environment variables to 4 or less from cmd. = How to do this properly? I get 'IPEX_ATTENTION_SLICE_RATE' is not recognized as an internal or external command, operable program or batch file.

And try disabling IPEX Optimize from settings. It might be bad on Windows. = It didn't work.

Disty0 commented 6 months ago

set ENV_VAR=value is the command on Windows. This needs to be applied every time tho.

Use Dynamic Attention SDP from Compute Settings if you don't want to mess with command line. Changing Attention Optimization method and IPEX Optimize needs model reload.

AndrewRainfall commented 6 months ago

Use Dynamic Attention SDP from Compute Settings

Found it, reloaded model & restarted SD Next. Reducing it to any value up to 0.1GB didn't work. But it significantly reduced image quality & it seems even increased chances of generating all images black, so I reverted it to 4GB and Scaled-Dot-Product.

set ENV_VAR=value is the command on Windows

Like in the image? If yes - it didn't work. download

Disty0 commented 6 months ago

Is this only happens with 512x512 x4? 512x512 x4 will be the same as 1024x1024 and Intel ARC has 1024x1024 curse. Try any other resolution or batch size.

AndrewRainfall commented 6 months ago

It happens with any resolution & any batch size. I get one image at average at any batch size up to 32

AndrewRainfall commented 6 months ago

Idk why, but it's fixed now. Hope it won't be back.

I'm sure I didn't do anything that could fix it since my last SD Next launch.

Thank you for helping me Disty0

Working log output, mb it'll help someone:

Using VENV: C:\automatic\venv 22:08:39-550302 INFO Starting SD.Next 22:08:39-556803 INFO Logger: file="C:\automatic\sdnext.log" level=INFO size=1030603 mode=append 22:08:39-560304 INFO Python 3.10.6 on Windows 22:08:39-795335 INFO Version: app=sd.next updated=2024-02-23 hash=c5f6c25a url=https://github.com/vladmandic/automatic/tree/master 22:08:40-500474 INFO Latest published version: c1dfb1b28e37f286107b44740ccf6d1733233992 2024-02-24T13:23:08Z 22:08:40-519478 INFO Platform: arch=AMD64 cpu=AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD system=Windows release=Windows-10-10.0.19045-SP0 python=3.10.6 22:08:40-527979 INFO Intel OneAPI Toolkit detected 22:08:40-702166 INFO Extensions: disabled=[] 22:08:40-705165 INFO Extensions: enabled=['Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg'] extensions-builtin 22:08:40-718164 INFO Extensions: enabled=[] extensions 22:08:40-720665 INFO Startup: quick launch 22:08:40-722165 INFO Verifying requirements 22:08:40-740168 INFO Verifying packages 22:08:40-744169 INFO Extensions: disabled=[] 22:08:40-747169 INFO Extensions: enabled=['Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg'] extensions-builtin 22:08:40-758171 INFO Extensions: enabled=[] extensions 22:08:40-769674 INFO Command line args: ['--use-ipex', '--autolaunch'] autolaunch=True use_ipex=True 22:08:52-695328 INFO Load packages: {'torch': '2.1.0a0+cxx11.abi', 'diffusers': '0.26.3', 'gradio': '3.43.2'} 22:08:54-056425 INFO Engine: backend=Backend.DIFFUSERS compute=ipex device=xpu attention="Scaled-Dot-Product" mode=no_grad 22:08:54-061424 INFO Device: device=Intel(R) Arc(TM) A770 Graphics n=1 ipex=2.1.10+xpu 22:08:55-749083 INFO Available VAEs: path="E:\models\VAE" items=3 22:08:55-752584 INFO Disabled extensions: ['sd-webui-controlnet'] 22:08:55-762572 INFO Available models: path="E:\models\Stable-diffusion" items=21 time=0.01 22:08:55-962588 INFO Extension: script='extensions-builtin\Lora\scripts\lora_script.py' 22:08:55-955587 INFO LoRA networks: available=17 folders=2 22:08:56-594514 INFO Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py' Using sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3 22:08:57-039681 INFO UI theme: name="black-teal" style=Dark base=sdnext.css 22:08:58-954237 INFO Extension list is empty: refresh required 22:09:00-127698 INFO Local URL: http://127.0.0.1:7860/ 22:09:00-621784 INFO [AgentScheduler] Task queue is empty 22:09:00-624285 INFO [AgentScheduler] Registering APIs 22:09:00-812819 INFO Select: refiner="cyberrealistic_v41BackToBasics [41b6846108]" 22:09:01-220779 INFO Setting Torch parameters: device=xpu dtype=torch.bfloat16 vae=torch.bfloat16 unet=torch.bfloat16 context=no_grad fp16=True bf16=True optimization=Scaled-Dot-Product 22:09:01-225780 INFO Loading VAE: model=E:\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors source=settings 22:09:01-228280 INFO Autodetect: vae="Stable Diffusion" class=StableDiffusionPipeline file="E:\models\Stable-diffusion\cyberrealistic_v41BackToBasics.safetensors" size=2034MB 22:09:01-528834 INFO Autodetect: refiner="Stable Diffusion" class=StableDiffusionPipeline file="E:\models\Stable-diffusion\cyberrealistic_v41BackToBasics.safetensors" size=2034MB 22:09:05-679780 INFO Load refiner: time=4.61 load=4.61 native=512 {'ram': {'used': 5.29, 'total': 31.95}, 'gpu': {'used': 0.0, 'total': 15.54}, 'retries': 0, 'oom': 0} 22:09:05-687781 INFO Startup time: 24.91 torch=10.03 olive=0.12 gradio=1.77 libraries=2.99 extensions=1.08 face-restore=0.12 ui-en=0.38 ui-txt2img=0.08 ui-img2img=0.10 ui-control=0.13 ui-settings=0.47 ui-extensions=1.46 ui-defaults=0.12 launch=0.22 api=0.15 app-started=0.53 checkpoint=4.88 22:09:05-700283 INFO Launching browser 22:09:08-254239 INFO MOTD: N/A 22:09:13-208332 INFO Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 22:09:54-375548 INFO Select: model="cyberrealistic_v41BackToBasics [41b6846108]" 22:09:54-384049 INFO Setting Torch parameters: device=xpu dtype=torch.bfloat16 vae=torch.bfloat16 unet=torch.bfloat16 context=no_grad fp16=True bf16=True optimization=Scaled-Dot-Product 22:09:54-389050 INFO Loading VAE: model=E:\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors source=settings 22:09:54-392051 INFO Autodetect: vae="Stable Diffusion" class=StableDiffusionPipeline file="E:\models\Stable-diffusion\cyberrealistic_v41BackToBasics.safetensors" size=2034MB 22:09:54-719608 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="E:\models\Stable-diffusion\cyberrealistic_v41BackToBasics.safetensors" size=2034MB 22:10:02-059572 INFO Load embeddings: loaded=7 skipped=1 time=0.86 22:10:02-317617 INFO Load model: time=7.68 load=7.68 native=512 {'ram': {'used': 5.08, 'total': 31.95}, 'gpu': {'used': 2.11, 'total': 15.54}, 'retries': 0, 'oom': 0}