search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
17.04k stars 1.96k forks source link

[Bug]: inpaint_only+lama preprocessor fails on DirectML #1630

Closed TeutonJon78 closed 11 months ago

TeutonJon78 commented 1 year ago

Is there an existing issue for this?

What happened?

The inpaint_only+lama preprocessor fails on the DirectML builds of both WebUI and SD.Next. The inpaint only preprocessor works correctly.

It errors out with:

[F D:\a\_work\1\s\pytorch-directml-plugin\torch_directml\csrc\engine\dml_util.cc:118] Invalid or unsupported data type.
Press any key to continue . . .

And then crashes the whole SD process.

Steps to reproduce the problem

  1. enable ControlNet on txt2img tab, enable preview
  2. drag an image into the image tab
  3. select inpaint_only for preprocessor, control_v11p_sd15_inpaint [ebff9138] for model
  4. click on preview button
  5. proprocessor completes correctly
  6. select inpaint_only+lama for preprocessor, control_v11p_sd15_inpaint [ebff9138] for model
  7. click on preview button
  8. error outputted and SD process crashes

What should have happened?

The preprocessor should complete and allow image generation.

Commit where the problem happens

webui (DirectML fork): https://github.com/lshqqytiger/stable-diffusion-webui-directml/commit/ba780a8e4fb27d1eb9595947380a63bd84fc419e SD.Next: https://github.com/vladmandic/automatic/commit/1afb7c6ca86c1e154f811e61da3f7c5ce0ae867f controlnet: v1.1.224 - https://github.com/Mikubill/sd-webui-controlnet/commit/c488ef2cbb4c02802564600abb0057fabd3d4cad

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--autolaunch --update-all-extensions --medvram --disable-nan-check  --precision full --no-half --no-half-vae

(Using Doggettx/opt-split-attention, also directory flags)

List of enabled extensions

a1111-sd-webui-lycoris canvas-zoom multidiffusion-upscaler-for-automatic1111 openpose-editor sd-dynamic-thresholding sd-extension-system-info sd-webui-3d-open-pose-editor sd-webui-aspect-ratio-helper sd-webui-controlnet sd-webui-infinite-image-browsing sd-webui-regional-prompter sd-webui-segment-anything sd_save_intermediate_images stable-diffusion-webui-composable-lora stable-diffusion-webui-images-browser stable-diffusion-webui-two-shot (ashen-sensored) ultimate-upscale-for-automatic1111

Console logs

venv "D:\Graphic Design\AI\stable-diffusion-webui-directml\venv\Scripts\Python.exe"
fatal: No names found, cannot describe anything.
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: <none>
Commit hash: ba780a8e4fb27d1eb9595947380a63bd84fc419e
Installing requirements

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\a1111-sd-webui-lycoris':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\canvas-zoom':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\multidiffusion-upscaler-for-automatic1111':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\openOutpaint-webUI-extension':
Already up to date.

You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

Couldn't perform 'git pull' on repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\openOutpaint-webUI-extension\app':

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\openpose-editor':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-dynamic-thresholding':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-extension-system-info':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-webui-3d-open-pose-editor':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-webui-aspect-ratio-helper':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-webui-controlnet':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-webui-infinite-image-browsing':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-webui-regional-prompter':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-webui-segment-anything':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd_save_intermediate_images':
Already up to date.

error: Pulling is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
Couldn't perform 'git pull' on repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\Stable-Diffusion-Webui-Civitai-Helper':

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\stable-diffusion-webui-composable-lora':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\stable-diffusion-webui-images-browser':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\stable-diffusion-webui-two-shot':
Already up to date.

Pulled changes for repository in 'D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\ultimate-upscale-for-automatic1111':
Already up to date.

Launching Web UI with arguments: --autolaunch --update-all-extensions --styles-file D:\Graphic Design\AI\styles\styles.csv --medvram --disable-nan-check --precision full --no-half --no-half-vae --ckpt-dir D:\Graphic Design\AI\SD models\Stable-diffusion --vae-dir D:\Graphic Design\AI\SD models\VAE --codeformer-models-path D:\Graphic Design\AI\SD models\Codeformer --esrgan-models-path D:\Graphic Design\AI\SD models\ESRGAN --gfpgan-models-path D:\Graphic Design\AI\SD models\GFPGAN --bsrgan-models-path D:\Graphic Design\AI\SD models\BSRGAN --realesrgan-models-path D:\Graphic Design\AI\SD models\RealESRGAN --scunet-models-path D:\Graphic Design\AI\SD models\ScuNET --swinir-models-path D:\Graphic Design\AI\SD models\SwinIR --ldsr-models-path D:\Graphic Design\AI\SD models\LDSR --lora-dir D:\Graphic Design\AI\SD models\Lora --lyco-dir D:\Graphic Design\AI\SD models\LyCORIS --clip-models-path D:\Graphic Design\AI\SD models\CLIP --embeddings-dir D:\Graphic Design\AI\SD models\embeddings --hypernetwork-dir D:\Graphic Design\AI\SD models\hypernetworks --controlnet-dir D:\Graphic Design\AI\SD models\ControlNet
No module 'xformers'. Proceeding without it.
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
2023-06-12 17:48:45,173 - ControlNet - INFO - ControlNet v1.1.224
ControlNet preprocessor location: D:\Graphic Design\AI\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\downloads
2023-06-12 17:48:45,270 - ControlNet - INFO - ControlNet v1.1.224
Image Browser: ImageReward is not installed, cannot be used.
Loading weights [2336dbf342] from D:\Graphic Design\AI\SD models\Stable-diffusion\dreamshaper_631BakedVae.safetensors
Creating model from config: D:\Graphic Design\AI\stable-diffusion-webui-directml\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: D:\Graphic Design\AI\SD models\VAE\vae-ft-mse-840000-ema-pruned.safetensors
Applying optimization: Doggettx... done.
Textual inversion embeddings loaded(18): advntr, AS-Adult, AS-Adult-neg, AS-Elderly, AS-MidAged, AS-Young, AS-Young-neg, AS-Younger, AS-Youngest, AS-YoungV2, AS-YoungV2-neg, charturnerv2, coloring-book-style, easynegative, olfn, OVERMASKED, PlanIt
Textual inversion embeddings skipped(6): 21charturnerv2, mdjrny-ppc, nartfixer, nfixer, nrealfixer, PlanIt2
Model loaded in 2.7s (load weights from disk: 1.2s, create model: 0.4s, apply weights to model: 0.8s, load VAE: 0.2s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 12.1s (import torch: 1.9s, import gradio: 1.3s, import ldm: 0.6s, other imports: 0.9s, load scripts: 2.8s, create ui: 3.4s, gradio launch: 0.9s).
2023-06-12 17:52:56,864 - ControlNet - INFO - Pixel Perfect Computation:
2023-06-12 17:52:56,865 - ControlNet - INFO - resize_mode = ResizeMode.INNER_FIT
2023-06-12 17:52:56,865 - ControlNet - INFO - raw_H = 512
2023-06-12 17:52:56,865 - ControlNet - INFO - raw_W = 512
2023-06-12 17:52:56,865 - ControlNet - INFO - target_H = 512
2023-06-12 17:52:56,865 - ControlNet - INFO - target_W = 512
2023-06-12 17:52:56,865 - ControlNet - INFO - estimation = 512.0
2023-06-12 17:52:56,866 - ControlNet - INFO - Preview Resolution = 512
[F D:\a\_work\1\s\pytorch-directml-plugin\torch_directml\csrc\engine\dml_util.cc:118] Invalid or unsupported data type.
Press any key to continue . . .


### Additional information

_No response_
TeutonJon78 commented 1 year ago

Only other reference I could find with the same issue (and the person posting it would make sense): https://github.com/microsoft/DirectML/issues/421

@lshqqytiger do you know if this is the same issue?

lshqqytiger commented 1 year ago

It seems like a similar issue, but I'm not sure it is the same issue. It requires a specific data type (is it complex? or other one?), but DirectML does not have support of that. When I found DirectML does not have complex type, I added print to most of the lines of the whole code and catched the specific point when that error occurred. Because it does not give us any traceback, I had to trace it back myself.

upright2003 commented 1 year ago

Hello! I followed this video tutorial and the same problem occurred. How to solve it? https://www.youtube.com/watch?v=BsxVtu3ocsc

2023-11-04 19:54:27,364 - ControlNet - INFO - preprocessor resolution = -1 [F D:\a_work\1\s\pytorch-directml-plugin\torch_directml\csrc\engine\dml_util.cc:118] Invalid or unsupported data type. Press any key to continue . . .

001 003 004

Lumpsack commented 1 year ago

I'm not 100% sure, but I think the issue is with DirectML, that falls to MS so don't expect a fix. I'm saddened by this too, it looks like a great solution, I wish I'd spent a bit more and gone nVidea, so many great solutions seem to be in the way under AMD.

theatnelson commented 12 months ago

Not sure if this helps but I literally had the exact same error:

[F D:\a\_work\1\s\pytorch-directml-plugin\torch_directml\csrc\engine\dml_util.cc:118] Invalid or unsupported data type.
Press any key to continue . . .

TLDR: Try removing all models from models\Stable-diffusion and rerun. This fixed the issue for me at least.

I was about to do a fresh install and moved all my downloaded and custom models out of models\Stable-diffusion to a different directory outside of the project just so i didn't have to redownload/remerge. I only left the "original v1-5-pruned-emaonly.safetensors" in the directory. Out of curiosity I reran the webui bat file and voila SD started right up with no issues.

It also looks like it was only the model Deliberate V3 that SD didn't like since it worked with every other model even custom merged ones. Funny thing is that I can still add in Deliberate after SD boot and apply the model with no issue, I just can't have it in the model dir at startup.

Hope this helps.

ride5k commented 9 months ago

I'm not 100% sure, but I think the issue is with DirectML, that falls to MS so don't expect a fix. I'm saddened by this too, it looks like a great solution, I wish I'd spent a bit more and gone nVidea, so many great solutions seem to be in the way under AMD.

this is unfortunate, as i am getting the same error on AMD when trying to use inpaint_only+lama

BrothaPitagoras commented 5 months ago

image Exact same error, would've been great to have a solution, saddened that I bought an AMD card without thinking about SD, so many workarounds for what previous to having an AMD GPU was as simple as installing and running.

lshqqytiger commented 5 months ago

Please use ZLUDA that has complex data type support if possible.

BrothaPitagoras commented 5 months ago

Please use ZLUDA that has complex data type support if possible.

After installing zluda I can’t seem to find models that work on it, all models say “model not loaded” upon trying to generate and somehow control net is installed but I don’t see it in Zluda’s cluttered and contrasty UI, I’ll look into it more since it seems promising but thus far it’s problem galore