[Bug]: (Embedding) Training suddenly not working anymore

Woisek commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits

What happened?

As I tried to do a new training for an embedding, an error occured.

Steps to reproduce the problem

Setup embedding training
Start training

What should have happened?

Embedding training should start and work.

Commit where the problem happens

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/ea9bd9fc7409109adcd61b897abc2c8881161256

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--xformers --api --administrator --opt-split-attention --theme=dark

List of extensions

Console logs

Already up to date.
venv "I:\Super SD 2.0\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Commit hash: ea9bd9fc7409109adcd61b897abc2c8881161256
Installing requirements for Web UI
loading Smart Crop reqs from I:\Super SD 2.0\stable-diffusion-webui\extensions\sd_smartprocess\requirements.txt
Checking Smart Crop requirements.

Launching Web UI with arguments: --xformers --api --administrator --opt-split-attention --theme=dark
==============================================================================
You are running torch 1.12.1+cu113.
The program is tested to work with torch 1.13.1.
To reinstall the desired version, run with commandline flag --reinstall-torch.
Beware that this will cause a lot of large files to be downloaded, as well as
there are reports of issues with training tab on the latest version.

Use --skip-version-check commandline argument to disable this check.
==============================================================================
=================================================================================
You are running xformers 0.0.14.dev.
The program is tested to work with xformers 0.0.16rc425.
To reinstall the desired version, run with commandline flag --reinstall-xformers.

Use --skip-version-check commandline argument to disable this check.
=================================================================================
Loading weights [cc6cb27103] from I:\Super SD 2.0\stable-diffusion-webui\models\Stable-diffusion\SD v1.5\v1-5-pruned-emaonly_(3.97GB).ckpt
Creating model from config: I:\Super SD 2.0\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(107): a-tale-of-two-empires, abstract-concepts, alberto-montt, anders-zorn, angus-mcbride-emb, angus-mcbride-style, arcimboldo-emb, arcimboldo-style, baluchitherian, barbosa, black-and-white-design, boissonnard, cute-game-style, david-moreno-architecture, design, dreams, durer-style, egorey, elegant-flower, Exodus-Styling, fairy-tale-painting-style, FightsInTiSpa-emb, fractal-flame, frank-frazetta, franz-unterberger, fzk, green-blue-shanshui, gt-color-paint-2, guttestreker, hrgiger-drmacabre, huang-guang-jian, liquid-light, low-poly-hd-logos-icons, marbling-art, Marc_Allante, mass, maurice-quentin-de-la-tour-style, mikako-method, mikako-methodi2i, nebula, orientalist-art, painting, pastelartstyle, piotr-jablonski, pjablonski-style, PlanIt, prsnCaroKeb, ralph-mcquarrie, realMichaela_v1, realMichaela_v4-3500, realMichaela_v4, realRoman_v2-2200, realRoman_v2-3500, roy-lichtenstein, sherhook-painting-v2, sherhook-painting, shev-linocut, Style-Autumn, Style-Empire, Style-Glorious, Style-Necromancy, Style-Winter, summie-style, teelip-ir-landscape, tron-style, vkuoo1, wheatland-arknight, xyz, prsnAliLarter-3000, prsnAliLarter-4000, prsnAliLarter-4500, prsnAliLarter-4750, prsnAliLarter-6500, prsnAliLarter-6750, prsnAliLarter-7500, prsnAliLarter-8000, prsnAlmaZadic, Bob Ross-200, Bob Ross-2000, Bob Ross-800, Bob Ross-8000, style-bobross-1150, style-bobross-1400, style-bobross-2850, style-bobross-4950, style-bobross-500, style-bobross-5900, prsnKaroEtstad, prsnKaroEdst-1200_7, prsnKaroEdst-3120_9, prsnKaroEdst-3360_6, prsnKaroEdst-4080_9, prsnKaroEdst-5040_9, prsnKaroEdst-720_5, prsnKaroEdst-960_7, kc16-1000-sd1-5, kc16-2000-sd1-5, kc16-3000-sd1-5, kc16-4000-sd1-5, Masaccio, style-masaccio, style-masaccio2-8990, style-masaccio2-9920, prsnSpencerLocke-7200, prsnSpencerLocke-7440, prsnSpencerLocke-7680, prsnSpencerLocke-7920
Textual inversion embeddings skipped(34): anthro, bones, classipeint, GTA768, HorrorByDave, jozef-tominc2, lavastyle-12000, matej-sternen, mdjrny-ppc_SDv2, midjourney-style, Neg_Facelift768, painted_abstract, painting-made-by-bruegel-v4, papercutcraft-SDv2, ParchArt, PlanIt2, PortraitHelper, remix, ScaryMonstersV2, Style-Winter2, Unddep, VikingPunk, VintageHelper, vray-render, waterworksv2-5000, webui, CGI_Animation-185, CGI_Animation-245, CGI_Animation, kc16-v2-5000, kc16-v3-6250, kc16-v4-5000, kc32-v4-5000, kc8-v4-5000
Model loaded in 7.0s (load weights from disk: 2.8s, create model: 0.4s, apply weights to model: 0.6s, apply half(): 0.5s, move model to device: 0.5s, load textual inversion embeddings: 2.2s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:05<00:00,  3.47it/s]
Loading CLiP model ViT-L/14 ████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  9.27it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.59it/s]
Training at rate of 0.005 until step 100
Preparing dataset...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 58/58 [00:03<00:00, 17.42it/s]
No saved optimizer exists in checkpoint
  0%|                                                                                                              | 0/4000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "I:\Super SD 2.0\stable-diffusion-webui\modules\textual_inversion\textual_inversion.py", line 506, in train_embedding
    scaler.step(optimizer)
  File "I:\Super SD 2.0\stable-diffusion-webui\venv\lib\site-packages\torch\cuda\amp\grad_scaler.py", line 336, in step
    assert len(optimizer_state["found_inf_per_device"]) > 0, "No inf checks were recorded for this optimizer."
AssertionError: No inf checks were recorded for this optimizer.

Additional information

I trained an embedding in the forenoon this day without any problems.

Awethon commented 1 year ago

Do you find any of these suggestions helpful? https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5383

Woisek commented 1 year ago

Do you find any of these suggestions helpful? #5383

Many thanks for that hint. Indeed I found the (odd?) solution: I also had only used [name] for the prompt template, just because I thought it will giver better previews. Maybe someone shoul make a report on this, as it is (at least for me) not comprehensible why this is happening.

Anyway, thanks again for this note!

AUTOMATIC1111 / stable-diffusion-webui