search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
141.56k stars 26.75k forks source link

[Bug]: Creating Embedding on SDXL 1.0 base checkpoint #12165

Closed psynowczyk closed 1 year ago

psynowczyk commented 1 year ago

Is there an existing issue for this?

What happened?

There is an error when trying to create a new embedding while SDXL checkpoint is enabled.

Steps to reproduce the problem

  1. Select SDXL 1.0 base checkpoint
  2. Go to Train tab
  3. Try to create new embedding
  4. Observe errors in cmd

What should have happened?

New embedding should be created without any errors

Version or Commit where the problem happens

1.51

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

Cross attention optimization

xformers

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--port 7861 --xformers

List of extensions

Console logs

venv "D:\Programy\StableDiffusion\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: v1.5.1
Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a

Launching Web UI with arguments: --port 7861 --xformers
2023-07-29 18:16:08,082 - ControlNet - INFO - ControlNet v1.1.233
ControlNet preprocessor location: D:\Programy\StableDiffusion\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-07-29 18:16:08,239 - ControlNet - INFO - ControlNet v1.1.233
Loading weights [31e35c80fc] from D:\Programy\StableDiffusion\stable-diffusion-webui\models\Stable-diffusion\sdxl\base\sd_xl_base_1.0.safetensors
Running on local URL:  http://127.0.0.1:7861

To create a public link, set `share=True` in `launch()`.
Startup time: 17.6s (launcher: 8.1s, import torch: 3.4s, import gradio: 1.2s, setup paths: 0.8s, other imports: 1.0s, load scripts: 2.1s, create ui: 0.5s, gradio launch: 0.3s).
Creating model from config: D:\Programy\StableDiffusion\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Loading VAE weights specified in settings: D:\Programy\StableDiffusion\stable-diffusion-webui\models\VAE\sdxl_vae.safetensors
Applying attention optimization: xformers... done.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
No Image data blocks found.
Model loaded in 18.7s (load weights from disk: 1.3s, create model: 0.8s, apply weights to model: 7.3s, apply half(): 3.0s, load VAE: 0.6s, move model to device: 2.9s, load textual inversion embeddings: 2.1s, calculate empty prompt: 0.7s).
Traceback (most recent call last):
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1323, in process_api
    result = await self.call_function(
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1051, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\modules\textual_inversion\ui.py", line 11, in create_embedding
    filename = modules.textual_inversion.textual_inversion.create_embedding(name, nvpt, overwrite_old, init_text=initialization_text)
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\modules\textual_inversion\textual_inversion.py", line 283, in create_embedding
    cond_model([""])  # will send cond model to GPU if lowvram/medvram is active
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Programy\StableDiffusion\stable-diffusion-webui\repositories\generative-models\sgm\modules\encoders\modules.py", line 141, in forward
    emb_out = embedder(batch[embedder.input_key])
TypeError: list indices must be integers or slices, not str

Additional information

This issue occurs on SDXL 1.0 but not on 1.5 model

w-e-w commented 1 year ago

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/6f0abbb71a3f29d6df63fed82d5d5e196ca0d4de

psychonaut-s commented 1 year ago

6f0abbb

I've pulled these changes, but I'm still getting the same error as the original poster.

geoffpotter commented 1 year ago

Same, I tried the dev branch and the hires fix one. With and without xformers and/or no half vae.

I can't create embedings on the base model, but I can on the refiner model. Hypernetworks I can create on both.

Can't train either though, get these errors; ** Error training embedding Traceback (most recent call last): File "E:\stuff\stable-diffusion-webui\modules\textual_inversion\textual_inversion.py", line 532, in train_embedding loss = shared.sd_model.weighted_forward(x, cond, w)[0] / gradient_step File "E:\stuff\stable-diffusion-webui\modules\sd_hijack.py", line 132, in weighted_forward sd_model._old_get_loss = sd_model.get_loss File "E:\stuff\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1614, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'DiffusionEngine' object has no attribute 'get_loss'

`** Exception in training hypernetwork
Traceback (most recent call last):
  File "E:\stuff\stable-diffusion-webui\modules\hypernetworks\hypernetwork.py", line 619, in train_hypernetwork
    c = shared.sd_model.cond_stage_model(batch.cond_text).to(devices.device, non_blocking=pin_memory)
AttributeError: 'tuple' object has no attribute 'to'`
hansun11 commented 1 year ago

6f0abbb

I applied these changes ,but it is still the same problem. This is the log:

Traceback (most recent call last): File "E:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 422, in run_predict output = await app.get_blocks().process_api( File "E:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1323, in process_api result = await self.call_function( File "E:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1051, in call_function prediction = await anyio.to_thread.run_sync( File "E:\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "E:\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "E:\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run result = context.run(func, *args) File "E:\stable-diffusion-webui\modules\textual_inversion\ui.py", line 11, in create_embedding filename = modules.textual_inversion.textual_inversion.create_embedding(name, nvpt, overwrite_old, init_text=initialization_text) File "E:\stable-diffusion-webui\modules\textual_inversion\textual_inversion.py", line 293, in create_embedding cond_model([""]) # will send cond model to GPU if lowvram/medvram is active File "E:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "E:\stable-diffusion-webui\repositories\generative-models\sgm\modules\encoders\modules.py", line 141, in forward emb_out = embedder(batch[embedder.input_key]) TypeError: list indices must be integers or slices, not str

DoctorDerp commented 1 year ago

Hypernetworks I can create on both.

Wait, you can create hypernetworks on the SDXL model? How?

catboxanon commented 1 year ago

Train tab isn't planned to be supported for SDXL. https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/11757

train tab will not work.

auto also plans to deprecate the train tab in favor of an extension at some point. https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/11857#discussioncomment-6480804

Maintaining those training scripts is too exhausting of a process for me. I will not work on it and likely will do an update in some future version that removes the tab in favor of some extension. Everyone trains using kohya's scripts. Kohya's trained SD XL networks can be used in webui.

For reference, kohya's scripts do support embedding training for SDXL. https://github.com/kohya-ss/sd-scripts/blob/sdxl/train_textual_inversion.py

Closing.

DoctorDerp commented 1 year ago

For reference, kohya's scripts do support embedding training for SDXL.

That's for textual inversion, but what about hypernetworks? Is it safe to assume they are dead then? So unless someone comes along and makes an extension capable of hypernetwork training for SDXL?

catboxanon commented 1 year ago

I would suggest opening an issue in https://github.com/kohya-ss/sd-scripts if you still want to train hypernetworks. It supports training them, but I doubt SDXL support is going to work at the moment. https://github.com/kohya-ss/sd-scripts/blob/sdxl/finetune/hypernetwork_nai.py

My own two cents: they're a naive approach to finetuning (it requires training a entirely separate network, not really analogous to LoRA), and because they originate from the NovelAI leak documentation is very sparse around them.