search

d8ahazard / sd_dreambooth_extension

Other
1.85k stars 281 forks source link

[Bug]: Exception training model: [WinError 5] Access is denied #1371

Closed klubkelli closed 8 months ago

klubkelli commented 9 months ago

Is there an existing issue for this?

What happened?

When attempting to train a new model (or Lora) it fails with the error Exception training model: '[WinError 5] Access is denied: '

I'm on a windows machine with a Nvidia graphics card.

Steps to reproduce the problem

  1. Go to .... Dreambooth tab
  2. Press .... attempt to create a model or lora (any kind any steps)
  3. .That's it. There is no step three. IT fails.

Commit and libraries

Initializing Dreambooth Dreambooth revision: 1a1d1621086a4725fda1200256f319c845dc7a8a [!] xformers NOT installed. [+] torch version 2.0.1+cu118 installed. [+] torchvision version 0.15.2+cu118 installed. [+] accelerate version 0.23.0 installed. [+] diffusers version 0.21.4 installed. [+] transformers version 4.32.1 installed. [+] bitsandbytes version 0.41.1 installed.

Launching Web UI with arguments: No module 'xformers'. Proceeding without it. [2023-10-09 00:24:16,878][DEBUG][git.cmd] - Popen(['git', 'version'], cwd=D:\stable-diffusion-webui, universal_newlines=False, shell=None, istream=None) [2023-10-09 00:24:16,939][DEBUG][git.cmd] - Popen(['git', 'version'], cwd=D:\stable-diffusion-webui, universal_newlines=False, shell=None, istream=None) [2023-10-09 00:24:17,458][DEBUG][filelock] - Attempting to acquire lock 2169171614000 on D:\stable-diffusion-webui\cache.json.lock [2023-10-09 00:24:17,459][DEBUG][filelock] - Lock 2169171614000 acquired on D:\stable-diffusion-webui\cache.json.lock [2023-10-09 00:24:17,459][DEBUG][filelock] - Attempting to release lock 2169171614000 on D:\stable-diffusion-webui\cache.json.lock [2023-10-09 00:24:17,460][DEBUG][filelock] - Lock 2169171614000 released on D:\stable-diffusion-webui\cache.json.lock [2023-10-09 00:24:18,332][DEBUG][api.py] - SD-Webui API layer loaded Loading weights [4199bcdd14] from D:\stable-diffusion-webui\models\Stable-diffusion\revAnimated_v122.safetensors [2023-10-09 00:24:18,846][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.reference] - entering reference: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.html_block] - entering html_block: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.heading] - entering heading: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.lheading] - entering lheading: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,850][DEBUG][markdown_it.rules_block.paragraph] - entering paragraph: StateBlock(line=0,level=2,tokens=2), 0, 1, False [2023-10-09 00:24:18,926][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,927][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.reference] - entering reference: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.html_block] - entering html_block: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.heading] - entering heading: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.lheading] - entering lheading: StateBlock(line=0,level=0,tokens=0), 0, 1, False [2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.paragraph] - entering paragraph: StateBlock(line=0,level=0,tokens=0), 0, 1, False False

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local Creating model from config: D:\stable-diffusion-webui\configs\v1-inference.yaml LatentDiffusion: Running in eps-prediction mode

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local DiffusionWrapper has 859.52 M params. [2023-10-09 00:24:19,373][DEBUG][git.cmd] - Popen(['git', 'remote', 'get-url', '--all', 'origin'], cwd=D:\stable-diffusion-webui\extensions\sd_dreambooth_extension, universal_newlines=False, shell=None, istream=None) Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch(). [2023-10-09 00:24:19,645][DEBUG][api.py] - Loading Dreambooth API Endpoints. Startup time: 5.4s (import torch: 1.1s, import gradio: 0.5s, import ldm: 0.3s, other imports: 1.3s, load scripts: 1.2s, create ui: 0.7s, gradio launch: 0.2s). preload_extensions_git_metadata for 8 extensions took 0.56s Applying attention optimization: Doggettx... done. Textual inversion embeddings loaded(0): Model loaded in 3.4s (load weights from disk: 0.4s, create model: 0.8s, apply weights to model: 0.8s, apply half(): 0.7s, move model to device: 0.7s). [2023-10-09 00:25:40,801][DEBUG][git.cmd] - Popen(['git', 'fetch', '-v', '--dry-run', '--', 'origin'], cwd=D:\stable-diffusion-webui\extensions\sd_dreambooth_extension, universal_newlines=True, shell=None, istream=None) 100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:05<00:00, 3.92it/s] Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00, 5.80it/s] [2023-10-09 00:27:46,901][DEBUG][dreambooth.dataclasses.db_config] - Saving to D:\stable-diffusion-webui\models\dreambooth\GemmaHiles Custom model name is gemmahiles1 [2023-10-09 00:27:48,022][DEBUG][dreambooth.dataclasses.db_config] - Saving to D:\stable-diffusion-webui\models\dreambooth\GemmaHiles Initializing dreambooth training... [2023-10-09 00:27:48,108][DEBUG][dreambooth.train_dreambooth] - Adding 'get_velocity' method to DEISMultistepScheduler... [2023-10-09 00:27:48,108][DEBUG][dreambooth.train_dreambooth] - Adding 'get_velocity' method to UniPCMultistepScheduler... [2023-10-09 00:27:48,258][DEBUG][dreambooth.train_dreambooth] - Pretrained path: D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\working [2023-10-09 00:27:48,455][INFO][dreambooth.utils.gen_utils] - Nothing to generate. | 12/16 [00:00<00:00, 113.51it/s]

===================================BUG REPORT=================================== | 0/1 [00:00<?, ?it/s]

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local [2023-10-09 00:29:27,405][WARNING][dreambooth.optimization] - Exception importing 8bit AdamW: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Traceback (most recent call last): File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\optimization.py", line 594, in get_optimizer from bitsandbytes.optim import AdamW8bit File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes__init__.py", line 6, in from . import cuda_setup, utils, research File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research__init.py", line 1, in from . import nn File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\nn\init.py", line 1, in from .modules import LinearFP8Mixed, LinearFP8Global File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in from bitsandbytes.optim import GlobalOptimManager File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\optim\init__.py", line 6, in from bitsandbytes.cextension import COMPILED_WITH_CUDA File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\cextension.py", line 20, in raise RuntimeError(''' RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

    CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

WARNING: Using default optimizer (AdamW from Torch) [2023-10-09 00:29:27,553][DEBUG][dreambooth.utils.gen_utils] - Found 0 reg images. | 0/1 [00:00<?, ?it/s] [2023-10-09 00:29:27,554][DEBUG][dreambooth.utils.gen_utils] - Preparing dataset... Init dataset! Preparing Dataset (With Caching) Loading latent cache...h Caching): 0%| | 0/1 [00:00<?, ?it/s] Bucket 0 (512, 512, 0) - Instance Images: 16 | Class Images: 0 | Max Examples/batch: 16 Saving cache!ed latents...: 100%|███████████████████████████████████████████████████████████████| 16/16 [00:00<?, ?it/s] [2023-10-09 00:29:27,579][ERROR][dreambooth.dataset.db_dataset] - Error saving cache! Traceback (most recent call last): File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\dataset\db_dataset.py", line 165, in save_cache_file os.remove(cache_file) PermissionError: [WinError 5] Access is denied: 'D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\cache\cache_512.safetensors' Total Buckets 1 - Instance Images: 16 | Class Images: 0 | Max Examples/batch: 16

Total images / batch: 16, total examples: 16█████████████████████████████████████████| 16/16 [00:00<00:00, 1069.58it/s] [2023-10-09 00:29:27,594][DEBUG][dreambooth.utils.gen_utils] - Total dataset length (steps): 16 Initializing bucket counter! [2023-10-09 00:29:27,795][DEBUG][dreambooth.train_dreambooth] - Running training [2023-10-09 00:29:27,795][DEBUG][dreambooth.train_dreambooth] - Num batches each epoch = 4 [2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] - Num Epochs = 2 [2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] - Batch Size Per Device = 4 [2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] - Gradient Accumulation steps = 1 [2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Total train batch size (w. parallel, distributed & accumulation) = 4 [2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Text Encoder Epochs: 2 [2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Total optimization steps = 32 [2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Total training steps = 32 [2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Resuming from checkpoint: False [2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - First resume epoch: 0 [2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - First resume step: 0 [2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - Lora: True, Optimizer: 8bit AdamW, Prec: fp16 [2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - Gradient Checkpointing: True [2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - EMA: True [2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - UNET: True [2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] - Freeze CLIP Normalization Layers: False [2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] - LR (Lora): 0.0001 [2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] - Tenc LR (Lora): 5e-05 [2023-10-09 00:29:27,800][DEBUG][dreambooth.train_dreambooth] - V2: False Steps: 0%| | 0/32 [00:00<?, ?it/s]D:\stable-diffusion-webui\venv\lib\site-packages\diffusers\models\attention_processor.py:1468: FutureWarning: LoRAAttnProcessor2_0 is deprecated and will be removed in version 0.26.0. Make sure use AttnProcessor2_0 instead by settingLoRA layers to self.{to_q,to_k,to_v,to_out[0]}.lora_layer respectively. This will be done automatically when using LoraLoaderMixin.load_lora_weights deprecate( Steps: 100%|█████████████████████████████████████████| 32/32 [00:14<00:00, 3.07it/s, loss=0.0995, lr=6.6e-6, vram=5.8][2023-10-09 00:29:42,182][DEBUG][dreambooth.train_dreambooth] - Save completed/canceled. [2023-10-09 00:29:42,182][DEBUG][dreambooth.train_dreambooth] - Completed, enabling saves. D:\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead. warnings.warn( Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:08<00:00, 1.26s/it] [2023-10-09 00:29:51,704][DEBUG][dreambooth.utils.model_utils] - Enabled AttnProcessor2_0 for UNet2DConditionModel [2023-10-09 00:29:51,706][DEBUG][dreambooth.utils.model_utils] - Enabled AttnProcessor2_0 for AutoencoderKL, 1.75s/it] [2023-10-09 00:29:51,706][DEBUG][dreambooth.train_dreambooth] - Save image is set. [2023-10-09 00:29:51,707][DEBUG][dreambooth.train_dreambooth] - Loras dir: D:\stable-diffusion-webui\models\Lora [2023-10-09 00:29:51,707][DEBUG][dreambooth.train_dreambooth] - Saving lora to D:\stable-diffusion-webui\models\Lora\gemmahiles1_32.safetensors [2023-10-09 00:29:51,839][DEBUG][dreambooth.train_dreambooth] - Saving images... [2023-10-09 00:29:51,891][DEBUG][dreambooth.train_dreambooth] - Loading image pipeline from D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\working... Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:02<00:00, 2.38it/s] [2023-10-09 00:29:55,623][DEBUG][dreambooth.train_dreambooth] - Generating 0 samples... Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:02<00:00, 2.32it/s]Model name: GemmaHiles 0%| | 0/2 [00:00<?, ?it/s] D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\helpers\log_parser.py:324: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead. all_df_loss = all_df_loss.fillna(method="ffill") Saving D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\logging\loss_plot_0.png Saving D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\logging\ram_plot_0.png Cleanup log parse. Traceback (most recent call last):████████████████████████ | 1/2 [00:00<00:00, 3.00it/s] File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 422, in run_predict output = await app.get_blocks().process_api( File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1326, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1251, in postprocess_data prediction_value = postprocess_update_dict( File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 434, in postprocess_update_dict prediction_value["value"] = block.postprocess(prediction_value["value"]) File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\components.py", line 4461, in postprocess file = self.pil_to_temp_file(img, dir=self.DEFAULT_TEMP_DIR) File "D:\stable-diffusion-webui\modules\ui_tempdir.py", line 55, in save_pil_to_file file_obj = tempfile.NamedTemporaryFile(delete=False, suffix=".png", dir=dir) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 559, in NamedTemporaryFile file = _io.open(dir, mode, buffering=buffering, File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 556, in opener fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 256, in _mkstemp_inner fd = _os.open(file, flags, 0o600) FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\Admin\AppData\Local\Temp\gradio\tmp0o0la6h4.png' Traceback (most recent call last): File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 730, in start_training result = main(class_gen_method=class_gen_method) File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1809, in main return inner_loop() File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator return function(batch_size, grad_size, prof, *args, **kwargs) File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1766, in inner_loop check_save(True) File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 976, in check_save save_weights( File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1380, in save_weights convert_diffusers_to_kohya_lora(lora_save_file, meta, args.lora_weight) File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\diff_lora_to_sd_lora.py", line 120, in convert_diffusers_to_kohya_lora os.remove(path) PermissionError: [WinError 5] Access is denied: 'D:\stable-diffusion-webui\models\Lora\gemmahiles1_32.safetensors' Steps: 100%|█████████████████████████████████████████| 32/32 [00:28<00:00, 1.10it/s, loss=0.0995, lr=6.6e-6, vram=5.8] Duration: 00:02:09 Generating Samples: 100%|████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.55it/s] [2023-10-09 00:29:57,795][DEBUG][dreambooth.utils.model_utils] - Restored system models. Duration: 00:02:10

Command Line Arguments

git pull
@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=

call webui.bat

Console logs

D:\stable-diffusion-webui>git pull
error: Your local changes to the following files would be overwritten by merge:
        requirements.txt
        requirements_versions.txt
Please commit your changes or stash them before you merge.
Aborting
Updating f865d3e1..5ef669de
venv "D:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.4.1
Commit hash: f865d3e11647dfd6c7b2cdf90dde24680e58acd8
Installing requirements
No module 'xformers'. Proceeding without it.
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: 1a1d1621086a4725fda1200256f319c845dc7a8a
[!] xformers NOT installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.23.0 installed.
[+] diffusers version 0.21.4 installed.
[+] transformers version 4.32.1 installed.
[+] bitsandbytes version 0.41.1 installed.

Launching Web UI with arguments:
No module 'xformers'. Proceeding without it.
[2023-10-09 00:24:16,878][DEBUG][git.cmd] - Popen(['git', 'version'], cwd=D:\stable-diffusion-webui, universal_newlines=False, shell=None, istream=None)
[2023-10-09 00:24:16,939][DEBUG][git.cmd] - Popen(['git', 'version'], cwd=D:\stable-diffusion-webui, universal_newlines=False, shell=None, istream=None)
[2023-10-09 00:24:17,458][DEBUG][filelock] - Attempting to acquire lock 2169171614000 on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,459][DEBUG][filelock] - Lock 2169171614000 acquired on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,459][DEBUG][filelock] - Attempting to release lock 2169171614000 on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,460][DEBUG][filelock] - Lock 2169171614000 released on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:18,332][DEBUG][api.py] - SD-Webui API layer loaded
Loading weights [4199bcdd14] from D:\stable-diffusion-webui\models\Stable-diffusion\revAnimated_v122.safetensors
[2023-10-09 00:24:18,846][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.reference] - entering reference: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.html_block] - entering html_block: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.heading] - entering heading: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.lheading] - entering lheading: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,850][DEBUG][markdown_it.rules_block.paragraph] - entering paragraph: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,926][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,927][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.reference] - entering reference: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.html_block] - entering html_block: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.heading] - entering heading: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.lheading] - entering lheading: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.paragraph] - entering paragraph: StateBlock(line=0,level=0,tokens=0), 0, 1, False
False

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Creating model from config: D:\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
DiffusionWrapper has 859.52 M params.
[2023-10-09 00:24:19,373][DEBUG][git.cmd] - Popen(['git', 'remote', 'get-url', '--all', 'origin'], cwd=D:\stable-diffusion-webui\extensions\sd_dreambooth_extension, universal_newlines=False, shell=None, istream=None)
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
[2023-10-09 00:24:19,645][DEBUG][api.py] - Loading Dreambooth API Endpoints.
Startup time: 5.4s (import torch: 1.1s, import gradio: 0.5s, import ldm: 0.3s, other imports: 1.3s, load scripts: 1.2s, create ui: 0.7s, gradio launch: 0.2s).
preload_extensions_git_metadata for 8 extensions took 0.56s
Applying attention optimization: Doggettx... done.
Textual inversion embeddings loaded(0):
Model loaded in 3.4s (load weights from disk: 0.4s, create model: 0.8s, apply weights to model: 0.8s, apply half(): 0.7s, move model to device: 0.7s).
[2023-10-09 00:25:40,801][DEBUG][git.cmd] - Popen(['git', 'fetch', '-v', '--dry-run', '--', 'origin'], cwd=D:\stable-diffusion-webui\extensions\sd_dreambooth_extension, universal_newlines=True, shell=None, istream=None)
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:05<00:00,  3.92it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  5.80it/s]
[2023-10-09 00:27:46,901][DEBUG][dreambooth.dataclasses.db_config] - Saving to D:\stable-diffusion-webui\models\dreambooth\GemmaHiles
Custom model name is gemmahiles1
[2023-10-09 00:27:48,022][DEBUG][dreambooth.dataclasses.db_config] - Saving to D:\stable-diffusion-webui\models\dreambooth\GemmaHiles
Initializing dreambooth training...
[2023-10-09 00:27:48,108][DEBUG][dreambooth.train_dreambooth] - Adding 'get_velocity' method to DEISMultistepScheduler...
[2023-10-09 00:27:48,108][DEBUG][dreambooth.train_dreambooth] - Adding 'get_velocity' method to UniPCMultistepScheduler...
                  [2023-10-09 00:27:48,258][DEBUG][dreambooth.train_dreambooth] - Pretrained path: D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\working
                                                                                                                       [2023-10-09 00:27:48,455][INFO][dreambooth.utils.gen_utils] - Nothing to generate.     | 12/16 [00:00<00:00, 113.51it/s]

===================================BUG REPORT===================================                 | 0/1 [00:00<?, ?it/s]
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
[2023-10-09 00:29:27,405][WARNING][dreambooth.optimization] - Exception importing 8bit AdamW:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\optimization.py", line 594, in get_optimizer
    from bitsandbytes.optim import AdamW8bit
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\__init__.py", line 1, in <module>
    from . import nn
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\nn\__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
WARNING: Using default optimizer (AdamW from Torch)
                                                                                                                       [2023-10-09 00:29:27,553][DEBUG][dreambooth.utils.gen_utils] - Found 0 reg images.                | 0/1 [00:00<?, ?it/s]
[2023-10-09 00:29:27,554][DEBUG][dreambooth.utils.gen_utils] - Preparing dataset...
Init dataset!
Preparing Dataset (With Caching)
                                                                                                                       Loading latent cache...h Caching):   0%|                                                          | 0/1 [00:00<?, ?it/s]
Bucket 0 (512, 512, 0) - Instance Images: 16 | Class Images: 0 | Max Examples/batch: 16
                                                                                                                       Saving cache!ed latents...: 100%|███████████████████████████████████████████████████████████████| 16/16 [00:00<?, ?it/s]
[2023-10-09 00:29:27,579][ERROR][dreambooth.dataset.db_dataset] - Error saving cache!
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\dataset\db_dataset.py", line 165, in save_cache_file
    os.remove(cache_file)
PermissionError: [WinError 5] Access is denied: 'D:\\stable-diffusion-webui\\models\\dreambooth\\GemmaHiles\\cache\\cache_512.safetensors'
Total Buckets 1 - Instance Images: 16 | Class Images: 0 | Max Examples/batch: 16

Total images / batch: 16, total examples: 16█████████████████████████████████████████| 16/16 [00:00<00:00, 1069.58it/s]
                                                                                                                       [2023-10-09 00:29:27,594][DEBUG][dreambooth.utils.gen_utils] - Total dataset length (steps): 16
                  Initializing bucket counter!
[2023-10-09 00:29:27,795][DEBUG][dreambooth.train_dreambooth] -   ***** Running training *****
[2023-10-09 00:29:27,795][DEBUG][dreambooth.train_dreambooth] -   Num batches each epoch = 4
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] -   Num Epochs = 2
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] -   Batch Size Per Device = 4
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] -   Gradient Accumulation steps = 1
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Total train batch size (w. parallel, distributed & accumulation) = 4
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Text Encoder Epochs: 2
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Total optimization steps = 32
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Total training steps = 32
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Resuming from checkpoint: False
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   First resume epoch: 0
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   First resume step: 0
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   Lora: True, Optimizer: 8bit AdamW, Prec: fp16
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   Gradient Checkpointing: True
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   EMA: True
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   UNET: True
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] -   Freeze CLIP Normalization Layers: False
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] -   LR (Lora): 0.0001
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] -   Tenc LR (Lora): 5e-05
[2023-10-09 00:29:27,800][DEBUG][dreambooth.train_dreambooth] -   V2: False
Steps:   0%|                                                                                    | 0/32 [00:00<?, ?it/s]D:\stable-diffusion-webui\venv\lib\site-packages\diffusers\models\attention_processor.py:1468: FutureWarning: `LoRAAttnProcessor2_0` is deprecated and will be removed in version 0.26.0. Make sure use AttnProcessor2_0 instead by settingLoRA layers to `self.{to_q,to_k,to_v,to_out[0]}.lora_layer` respectively. This will be done automatically when using `LoraLoaderMixin.load_lora_weights`
  deprecate(
Steps: 100%|█████████████████████████████████████████| 32/32 [00:14<00:00,  3.07it/s, loss=0.0995, lr=6.6e-6, vram=5.8][2023-10-09 00:29:42,182][DEBUG][dreambooth.train_dreambooth] -
Save completed/canceled.
[2023-10-09 00:29:42,182][DEBUG][dreambooth.train_dreambooth] - Completed, enabling saves.
                                                                                                                       D:\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
  warnings.warn(
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:08<00:00,  1.26s/it]
[2023-10-09 00:29:51,704][DEBUG][dreambooth.utils.model_utils] - Enabled AttnProcessor2_0 for UNet2DConditionModel
[2023-10-09 00:29:51,706][DEBUG][dreambooth.utils.model_utils] - Enabled AttnProcessor2_0 for AutoencoderKL,  1.75s/it]
[2023-10-09 00:29:51,706][DEBUG][dreambooth.train_dreambooth] - Save image is set.
[2023-10-09 00:29:51,707][DEBUG][dreambooth.train_dreambooth] - Loras dir: D:\stable-diffusion-webui\models\Lora
                                                                                                                       [2023-10-09 00:29:51,707][DEBUG][dreambooth.train_dreambooth] - Saving lora to D:\stable-diffusion-webui\models\Lora\gemmahiles1_32.safetensors
[2023-10-09 00:29:51,839][DEBUG][dreambooth.train_dreambooth] - Saving images...
[2023-10-09 00:29:51,891][DEBUG][dreambooth.train_dreambooth] - Loading image pipeline from D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\working...
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:02<00:00,  2.38it/s]
[2023-10-09 00:29:55,623][DEBUG][dreambooth.train_dreambooth] - Generating 0 samples...
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:02<00:00,  2.32it/s]Model name: GemmaHiles 0%|                                                                        | 0/2 [00:00<?, ?it/s]
D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\helpers\log_parser.py:324: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
  all_df_loss = all_df_loss.fillna(method="ffill")
Saving D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\logging\loss_plot_0.png
Saving D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\logging\ram_plot_0.png
Cleanup log parse.
                                                                                                                       Traceback (most recent call last):████████████████████████                                | 1/2 [00:00<00:00,  3.00it/s]
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1326, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1251, in postprocess_data
    prediction_value = postprocess_update_dict(
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 434, in postprocess_update_dict
    prediction_value["value"] = block.postprocess(prediction_value["value"])
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\components.py", line 4461, in postprocess
    file = self.pil_to_temp_file(img, dir=self.DEFAULT_TEMP_DIR)
  File "D:\stable-diffusion-webui\modules\ui_tempdir.py", line 55, in save_pil_to_file
    file_obj = tempfile.NamedTemporaryFile(delete=False, suffix=".png", dir=dir)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 559, in NamedTemporaryFile
    file = _io.open(dir, mode, buffering=buffering,
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 556, in opener
    fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 256, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Admin\\AppData\\Local\\Temp\\gradio\\tmp0o0la6h4.png'
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 730, in start_training
    result = main(class_gen_method=class_gen_method)
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1809, in main
    return inner_loop()
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator
    return function(batch_size, grad_size, prof, *args, **kwargs)
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1766, in inner_loop
    check_save(True)
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 976, in check_save
    save_weights(
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1380, in save_weights
    convert_diffusers_to_kohya_lora(lora_save_file, meta, args.lora_weight)
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\diff_lora_to_sd_lora.py", line 120, in convert_diffusers_to_kohya_lora
    os.remove(path)
PermissionError: [WinError 5] Access is denied: 'D:\\stable-diffusion-webui\\models\\Lora\\gemmahiles1_32.safetensors'
Steps: 100%|█████████████████████████████████████████| 32/32 [00:28<00:00,  1.10it/s, loss=0.0995, lr=6.6e-6, vram=5.8]
Duration: 00:02:09
Generating Samples: 100%|████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.55it/s]
[2023-10-09 00:29:57,795][DEBUG][dreambooth.utils.model_utils] - Restored system models.
Duration: 00:02:10

Additional information

Version v1.4.1 Python 3.10.6 Torch 2.0.1+Cu118 xformers: N/A Gradio: 3.32.0 Checkpoint: 4199bcdd14

klubkelli commented 9 months ago

So no update?

klubkelli commented 9 months ago

As an update (For what it's worth) I updated my Nvidia graphics drivers to the last available as a new one came out today 537.58 and still did not fix my issue. Same error.

klubkelli commented 9 months ago

This issue seems to also be related to the problems that others are having such as https://github.com/d8ahazard/sd_dreambooth_extension/issues/1369 and https://github.com/d8ahazard/sd_dreambooth_extension/issues/1372.

Rocksar commented 9 months ago

Hey just here to say that i have the exact same problem and still no solution (more than 5hour of digging solutions)

SilverRider76 commented 9 months ago

Same here, I've also try a clean installation. Nothing is working

noahjgreer commented 8 months ago

Also still having this problem. It's a bit of a pain as it limits me from saving checkpoints during generation. If I let it run through all the steps, it will give the error at the end but it will also still output the trained lora model to the models/Lora folder. Hopefully a fix for this issue will come out soon.

swumagic commented 8 months ago

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang） 1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

swumagic commented 8 months ago

OR you are Linux distribution (Ubuntu, MacOS, etc.)system ,AND CUDA Version: 11.X.

Bitsandbytes can support ubuntu.（yuhuang） 1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

swumagic commented 8 months ago

OR you are Linux distribution (Ubuntu, MacOS, etc.)system ,AND CUDA Version: 11.X.

Bitsandbytes can support ubuntu.（yuhuang） 1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

d8ahazard commented 8 months ago

Closed, duplicate of #1360 , #1369 , also fixed with latest release.