search

d8ahazard / sd_dreambooth_extension

Other
1.85k stars 281 forks source link

[Bug]: Problem with 8bit AdamW #1395

Closed CrAvErS closed 8 months ago

CrAvErS commented 8 months ago

Is there an existing issue for this?

What happened?

I can't use 8bit adamw as optimizer, error code:

WARNING:dreambooth.optimization:Exception importing 8bit AdamW: AdamW8bit.__init__() got an unexpected keyword argument 'is_paged'
Traceback (most recent call last):
  File "D:\AI\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\optimization.py", line 595, in get_optimizer
    return AdamW8bit(
TypeError: AdamW8bit.__init__() got an unexpected keyword argument 'is_paged'
AdamW8bit.__init__() got an unexpected keyword argument 'is_paged'
WARNING: Using default optimizer (AdamW from Torch)

Steps to reproduce the problem

  1. Go to dreambooth extension
  2. create model (v2x)
  3. select 8bit AdamW as optimizer
  4. Start train

Commit and libraries

Starting at Initializing Dreambooth and ending several lines below at [+] bitsandbytes version 0.35.4 installed..

Initializing Dreambooth Dreambooth revision: c548ede6a3acb85d78650ff50ae583786b305f0e Checking xformers... Checking bitsandbytes... Checking bitsandbytes (Windows) Virtual environment path: D:\AI\stable-diffusion-webui\venv Checking for D:\AI\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll Checking Dreambooth requirements... [Dreambooth] accelerate v0.21.0 is already installed. [Dreambooth] dadaptation v3.1 is already installed. [Dreambooth] diffusers v0.23.1 is already installed. [Dreambooth] discord-webhook v1.3.0 is already installed. [Dreambooth] fastapi is already installed. [Dreambooth] gitpython v3.1.40 is not installed. Successfully installed gitpython-3.1.40

[Dreambooth] pytorch_optimizer v2.12.0 is already installed. [Dreambooth] Pillow is already installed. [Dreambooth] tqdm is already installed. [Dreambooth] tomesd v0.1.2 is already installed. [Dreambooth] transformers v4.30.2 is already installed. [Dreambooth] tensorboard v2.13.0 is already installed. [+] xformers version 0.0.21 installed. [+] torch version 2.0.1+cu118 installed. [+] torchvision version 0.15.2+cu118 installed. [+] accelerate version 0.21.0 installed. [+] diffusers version 0.23.1 installed. [+] transformers version 4.30.2 installed. [!] bitsandbytes version 0.35.4 installed.

Command Line Arguments

--xformers --reinstall-xformers --no-half-vae --disable-nan-check

Console logs

venv "D:\AI\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.6.0
Commit hash: 5ef669de080814067961f28357256e8fe27544f4
Installing xformers
Installing requirements
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: c548ede6a3acb85d78650ff50ae583786b305f0e
Checking xformers...
Checking bitsandbytes...
Checking bitsandbytes (Windows)
Virtual environment path: D:\AI\stable-diffusion-webui\venv
Checking for D:\AI\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll
Checking Dreambooth requirements...
[Dreambooth] accelerate v0.21.0 is already installed.
[Dreambooth] dadaptation v3.1 is already installed.
[Dreambooth] diffusers v0.23.1 is already installed.
[Dreambooth] discord-webhook v1.3.0 is already installed.
[Dreambooth] fastapi is already installed.
[Dreambooth] gitpython v3.1.40 is not installed.
Successfully installed gitpython-3.1.40

[Dreambooth] pytorch_optimizer v2.12.0 is already installed.
[Dreambooth] Pillow is already installed.
[Dreambooth] tqdm is already installed.
[Dreambooth] tomesd v0.1.2 is already installed.
[Dreambooth] transformers v4.30.2 is already installed.
[Dreambooth] tensorboard v2.13.0 is already installed.
[+] xformers version 0.0.21 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.21.0 installed.
[+] diffusers version 0.23.1 installed.
[+] transformers version 4.30.2 installed.
[!] bitsandbytes version 0.35.4 installed.
Launching Web UI with arguments: --xformers --reinstall-xformers --no-half-vae --disable-nan-check
Loading weights [879db523c3] from D:\AI\stable-diffusion-webui\models\Stable-diffusion\dreamshaper_8.safetensors
CUDA SETUP: Loading binary D:\AI\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cudaall.dll...
Creating model from config: D:\AI\stable-diffusion-webui\configs\v1-inference.yaml
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 49.6s (prepare environment: 32.1s, import torch: 7.1s, import gradio: 1.9s, setup paths: 2.0s, initialize shared: 0.4s, other imports: 1.1s, setup codeformer: 0.3s, load scripts: 2.7s, create ui: 1.1s, gradio launch: 0.7s).
Applying attention optimization: xformers... done.
Model loaded in 5.2s (load weights from disk: 0.8s, create model: 1.5s, apply weights to model: 2.4s, calculate empty prompt: 0.2s).
Advanced elements visible: False
Initializing dreambooth training...
                                                                 We need a total of 14 class images.
Loading pipeline components...:  71%|█████████████████████████████████████▏              | 5/7 [00:04<00:01,  1.25it/s]D:\AI\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
  warnings.warn(
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:07<00:00,  1.02s/it]
Using scheduler: DEISMultistep
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  6.68it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.92it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.87it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.89it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.82it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:04<00:00,  8.06it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:04<00:00,  8.11it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.84it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:04<00:00,  8.14it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.86it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.98it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:04<00:00,  8.04it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.96it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:05<00:00,  7.99it/s]
Generating class images 13/14::  93%|████████████████████████████████████████████▌   | 13/14 [00:00<00:00, 1925.83it/s]WARNING:dreambooth.optimization:Exception importing 8bit AdamW: AdamW8bit.__init__() got an unexpected keyword argument 'is_paged'
Traceback (most recent call last):
  File "D:\AI\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\optimization.py", line 595, in get_optimizer
    return AdamW8bit(
TypeError: AdamW8bit.__init__() got an unexpected keyword argument 'is_paged'
AdamW8bit.__init__() got an unexpected keyword argument 'is_paged'
WARNING: Using default optimizer (AdamW from Torch)
                                                                                                                       Init dataset!set:   0%|                                                                           | 0/5 [00:00<?, ?it/s]
Preparing Dataset (With Caching)
                                                                                                                       Loading latent cache...h Caching):   0%|                                                          | 0/5 [00:00<?, ?it/s]
Bucket 0 (512, 512, 0) - Instance Images: 33 | Class Images: 165 | Max Examples/batch:  66
                                                                                                                       Saving cache!ed latents...: 100%|████████████████████████████████████████████████████| 198/198 [00:01<00:00, 290.35it/s]
Total Buckets 1 - Instance Images: 33 | Class Images: 165 | Max Examples/batch:  66

Total images / batch: 66, total examples: 66████████████████████████████████████████| 198/198 [00:01<00:00, 290.35it/s]
                  Initializing bucket counter!
Steps:   2%|  | 218/9900 [13:04<9:46:57,  3.64s/it, inst_loss=0.0127, loss=0.0127, lr=2e-6, prior_loss=None, vram=12.6]
0it [00:15, ?it/s]

Additional information

No response

CrAvErS commented 8 months ago

reinstalling bitsandbytes helped.