d8ahazard / sd_dreambooth_extension

Other
1.85k stars 281 forks source link

[Bug]: continue on Lora training not possible and other errors #1376

Closed madrooky closed 4 months ago

madrooky commented 9 months ago

Is there an existing issue for this?

What happened?

Yesterday I updated dreambooth and since then various thinks broke. I got it running again but there are still errors. But most importantly, I can't continue training Lora pt's. It tries to load it from a hugging face rep.... silly...

Might be able to change that manually? I am looking through the settings but I am not even sure where and what to look for.

Note, that I made a separate fresh install for troubleshooting. There is no problem in a fresh install until i install dreambooth and then things fall apart. Spent hours yesterday and today trying to fix it.

I just dropped the entire log as far as it reaches back from my current console. It runs but it is insane how utterly broken it is. And once i think i have solved one thing another one replaces it.

Steps to reproduce the problem

install SD, install dreambooth, try to use it in any way....

Commit and libraries

Dreambooth revision: 1a1d1621086a4725fda1200256f319c845dc7a8a [!] xformers version 0.0.20 installed. [+] torch version 2.0.1+cu118 installed. [+] torchvision version 0.15.2+cu118 installed. [+] accelerate version 0.23.0 installed. [+] diffusers version 0.21.4 installed. [+] transformers version 4.32.1 installed. [+] bitsandbytes version 0.41.1 installed.

Command Line Arguments

set COMMANDLINE_ARGS= --api --xformers --ckpt-dir "S:\SD ext\models" --embeddings-dir "S:\SD ext\embeddings" --cors-allow-origins="http://127.0.0.1:7860" --skip-python-version-check
set "REQS_FILE=extensions\sd_dreambooth_extension\requirements.txt"

Console logs

S:\SD_core\stable-diffusion-webui>webui-user.bat
venv "S:\SD_core\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)]
Version: v1.6.0
Commit hash: 5ef669de080814067961f28357256e8fe27544f4
Installing requirements
[Auto-Photoshop-SD] Attempting auto-update...
[Auto-Photoshop-SD] switch branch to extension branch.
checkout_result: Your branch is up to date with 'origin/master'.

[Auto-Photoshop-SD] Current Branch.
branch_result: * master

[Auto-Photoshop-SD] Fetch upstream.
fetch_result:
[Auto-Photoshop-SD] Pull upstream.
pull_result: Already up to date.
Launching Web UI with arguments: --api --xformers --ckpt-dir S:\SD ext\models --embeddings-dir S:\SD ext\embeddings --cors-allow-origins=http://127.0.0.1:7860 --skip-python-version-check
python_server_full_path:  S:\SD_core\stable-diffusion-webui\extensions\Auto-Photoshop-StableDiffusion-Plugin\server/python_server2023-10-27 08:40:09,659 - ControlNet - INFO - ControlNet v1.1.411
ControlNet preprocessor location: S:\SD_core\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-10-27 08:40:09,806 - ControlNet - INFO - ControlNet v1.1.411
Loading weights [caa95002a1] from S:\SD ext\models\dl inpaint\epicrealism_pureEvolutionV5-inpainting.safetensors
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Creating model from config: S:\SD_core\stable-diffusion-webui\configs\v1-inpainting-inference.yaml
Startup time: 34.1s (prepare environment: 22.7s, import torch: 3.4s, import gradio: 1.2s, setup paths: 0.8s, initialize shared: 0.2s, other imports: 3.0s, setup codeformer: 0.1s, load scripts: 1.6s, create ui: 0.4s, gradio launch: 0.6s).
Applying attention optimization: xformers... done.
Model loaded in 2.8s (load weights from disk: 0.7s, create model: 0.4s, apply weights to model: 1.4s, calculate empty prompt: 0.1s).
Restarting UI...
Closing server running on port: 7860
python_server_full_path:  S:\SD_core\stable-diffusion-webui\extensions\Auto-Photoshop-StableDiffusion-Plugin\server/python_server2023-10-27 08:41:26,518 - ControlNet - INFO - ControlNet v1.1.411
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 1.1s (load scripts: 0.5s, create ui: 0.3s, gradio launch: 0.1s).
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\uvicorn\protocols\websockets\websockets_impl.py", line 247, in run_asgi
    result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__      
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\fastapi\applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\errors.py", line 149, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\base.py", line 26, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\base.py", line 26, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\cors.py", line 76, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\gzip.py", line 26, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__       
    raise exc
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__       
    await self.app(scope, receive, sender)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__     
    raise e
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__     
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\routing.py", line 341, in handle
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\routing.py", line 82, in app
    await func(session)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\fastapi\routing.py", line 289, in app
    await dependant.call(**values)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\gradio\routes.py", line 619, in join_queue
    if blocks.dependencies[event.fn_index].get("every", 0):
       ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
IndexError: list index out of range
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\uvicorn\protocols\websockets\websockets_impl.py", line 247, in run_asgi
    result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__      
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\fastapi\applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\errors.py", line 149, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\base.py", line 26, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\base.py", line 26, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\cors.py", line 76, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\gzip.py", line 26, in __call__
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__       
    raise exc
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__       
    await self.app(scope, receive, sender)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__     
    raise e
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__     
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\routing.py", line 341, in handle
    await self.app(scope, receive, send)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\starlette\routing.py", line 82, in app
    await func(session)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\fastapi\routing.py", line 289, in app
    await dependant.call(**values)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\gradio\routes.py", line 619, in join_queue
    if blocks.dependencies[event.fn_index].get("every", 0):
       ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
IndexError: list index out of range
Python 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)]
Version: v1.6.0
Commit hash: 5ef669de080814067961f28357256e8fe27544f4
Installing requirements
[Auto-Photoshop-SD] Attempting auto-update...
[Auto-Photoshop-SD] switch branch to extension branch.
checkout_result: Your branch is up to date with 'origin/master'.

[Auto-Photoshop-SD] Current Branch.
branch_result: * master

[Auto-Photoshop-SD] Fetch upstream.
fetch_result:
[Auto-Photoshop-SD] Pull upstream.
pull_result: Already up to date.
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: 1a1d1621086a4725fda1200256f319c845dc7a8a
[!] xformers version 0.0.20 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.23.0 installed.
[+] diffusers version 0.21.4 installed.
[+] transformers version 4.32.1 installed.
[+] bitsandbytes version 0.41.1 installed.
Launching Web UI with arguments: --api --xformers --ckpt-dir S:\SD ext\models --embeddings-dir S:\SD ext\embeddings --cors-allow-origins=http://127.0.0.1:7860 --skip-python-version-check
python_server_full_path:  S:\SD_core\stable-diffusion-webui\extensions\Auto-Photoshop-StableDiffusion-Plugin\server/python_server2023-10-27 08:45:58,685 - ControlNet - INFO - ControlNet v1.1.411
ControlNet preprocessor location: S:\SD_core\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-10-27 08:45:58,836 - ControlNet - INFO - ControlNet v1.1.411
Loading weights [caa95002a1] from S:\SD ext\models\dl inpaint\epicrealism_pureEvolutionV5-inpainting.safetensors
S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\scripts\main.py:301: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  with gr.Row().style(equal_height=False):
False

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('7860" --skip-python-version-check'), WindowsPath('/SD ext/embeddings" --cors-allow-origins="http'), WindowsPath('/127.0.0.1'), WindowsPath('/SD ext/models" --embeddings-dir "S'), WindowsPath(' --api --xformers --ckpt-dir "S')}
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...   
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('7860" --skip-python-version-check'), WindowsPath('/SD ext/embeddings" --cors-allow-origins="http'), WindowsPath('/127.0.0.1'), WindowsPath('/SD ext/models" --embeddings-dir "S'), WindowsPath(' --api --xformers --ckpt-dir "S')}
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...   
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Creating model from config: S:\SD_core\stable-diffusion-webui\configs\v1-inpainting-inference.yaml

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('7860" --skip-python-version-check'), WindowsPath('/SD ext/embeddings" --cors-allow-origins="http'), WindowsPath('/127.0.0.1'), WindowsPath('/SD ext/models" --embeddings-dir "S'), WindowsPath(' --api --xformers --ckpt-dir "S')}
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...   
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('7860" --skip-python-version-check'), WindowsPath('/SD ext/embeddings" --cors-allow-origins="http'), WindowsPath('/127.0.0.1'), WindowsPath('/SD ext/models" --embeddings-dir "S'), WindowsPath(' --api --xformers --ckpt-dir "S')}
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...   
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 31.5s (prepare environment: 19.4s, import torch: 3.4s, import gradio: 1.2s, setup paths: 0.8s, initialize shared: 0.2s, other imports: 3.0s, setup codeformer: 0.1s, load scripts: 2.2s, create ui: 1.0s, gradio launch: 0.1s).
Applying attention optimization: xformers... done.
Model loaded in 2.8s (load weights from disk: 0.5s, create model: 0.5s, apply weights to model: 1.4s, calculate empty prompt: 0.1s).
Initializing dreambooth training...

===================================BUG REPORT===================================                          | 0/1 [00:00<?, ?it/s]
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('7860" --skip-python-version-check'), WindowsPath('/SD ext/embeddings" --cors-allow-origins="http'), WindowsPath('/127.0.0.1'), WindowsPath('/SD ext/models" --embeddings-dir "S'), WindowsPath(' --api --xformers --ckpt-dir "S')}
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...   
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
WARNING:dreambooth.optimization:Exception importing 8bit AdamW:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Traceback (most recent call last):
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\optimization.py", line 594, in get_optimizer
    from bitsandbytes.optim import AdamW8bit
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\research\__init__.py", line 1, in <module>
    from . import nn
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\research\nn\__init__.py", line 1, in <module>      
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in <module>       
    from bitsandbytes.optim import GlobalOptimManager
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
WARNING: Using default optimizer (AdamW from Torch)
                                                                                                                                Init dataset!set:   0%|                                                                                    | 0/1 [00:00<?, ?it/s] 
Preparing Dataset (Without Caching)
Bucket 0 (624, 936, 0) - Instance Images: 27 | Class Images: 0 | Max Examples/batch: 27
Bucket 1 (936, 624, 0) - Instance Images: 15 | Class Images: 0 | Max Examples/batch: 15
Total Buckets 2 - Instance Images: 42 | Class Images: 0 | Max Examples/batch: 42

Total images / batch: 42, total examples: 42█████████████████████████████████████████████████| 42/42 [00:00<00:00, 21001.52it/s] 
                  Initializing bucket counter!
Steps:   0%|                                                                                            | 0/210 [00:00<?, ?it/s]S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\diffusers\models\attention_processor.py:1468: FutureWarning: `LoRAAttnProcessor2_0` is deprecated and will be removed in version 0.26.0. Make sure use AttnProcessor2_0 instead by settingLoRA layers to `self.{to_q,to_k,to_v,to_out[0]}.lora_layer` respectively. This will be done automatically when using `LoraLoaderMixin.load_lora_weights`
  deprecate(
Steps: 100%|███████████████████████████████████████████████| 210/210 [05:55<00:00,  1.63s/it, loss=0.00587, lr=0.0001, vram=5.8]S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
  warnings.warn(
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████| 7/7 [00:03<00:00,  1.94it/s]
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████| 7/7 [00:03<00:00,  1.83it/s]
Saving Lora Weights...:   0%|                                                                             | 0/1 [00:00<?, ?it/s]Model name: testles:   0%|                                                                                 | 0/2 [00:00<?, ?it/s] 
S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\helpers\log_parser.py:324: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
  all_df_loss = all_df_loss.fillna(method="ffill")
Saving S:\SD_core\stable-diffusion-webui\models\dreambooth\test\logging\loss_plot_0.png
Saving S:\SD_core\stable-diffusion-webui\models\dreambooth\test\logging\ram_plot_0.png
Cleanup log parse.
                                                                                                                                Traceback (most recent call last):████████████████████████████▌                                    | 1/2 [00:00<00:00,  3.25it/s] 
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 730, in start_training
    result = main(class_gen_method=class_gen_method)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1809, in main 
    return inner_loop()
           ^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator       
    return function(batch_size, grad_size, prof, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1766, in inner_loop
    check_save(True)
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 976, in check_save
    save_weights(
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1380, in save_weights
    convert_diffusers_to_kohya_lora(lora_save_file, meta, args.lora_weight)
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\diff_lora_to_sd_lora.py", line 120, in convert_diffusers_to_kohya_lora
    os.remove(path)
PermissionError: [WinError 5] Zugriff verweigert: 'S:\\SD_core\\stable-diffusion-webui\\models\\Lora\\test_210.safetensors'      
Steps: 100%|███████████████████████████████████████████████| 210/210 [06:06<00:00,  1.74s/it, loss=0.00587, lr=0.0001, vram=5.8] 
Duration: 00:06:20
Generating Samples: 100%|█████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.22it/s]
Duration: 00:06:21
Lora path: 
Compiling checkpoint for test...
Converting vae...
Converting text encoder...
Converting text enc dict for V1 model.
Saving checkpoint to S:\SD ext\models\test_0.safetensors...
Compiling checkpoint: 100%|████████████████████████████████████████████████████████████| 1130/1130 [00:00<00:00, 1130351.42it/s] 
Compiling checkpoint: 100%|█████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1502.62it/s] 
Copying config file from S:\SD ext\models\test_0.yaml to S:\SD ext\models\test_0.yaml
Checkpoint compiled successfully: S:\SD ext\models\test_0.safetensors
Duration: 00:00:08
Reusing loaded model dl inpaint\epicrealism_pureEvolutionV5-inpainting.safetensors [caa95002a1] to load test_0.safetensors
Calculating sha256 for S:\SD ext\models\test_0.safetensors: 94531057c05f52d6645f3d82ba426f32b8c7ff1b8f206d6564a94de7f6515f53
Loading weights [94531057c0] from S:\SD ext\models\test_0.safetensors
Creating model from config: S:\SD ext\models\test_0.yaml
Applying attention optimization: xformers... done.
Model loaded in 2.4s (create model: 0.3s, apply weights to model: 1.9s).
100%|███████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:14<00:00,  1.38it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████| 20/20 [00:15<00:00,  1.31it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.60it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.62it/s]
Reusing loaded model test_0.safetensors [94531057c0] to load dl models\epicrealism_pureEvolutionV5.safetensors [76be5be1b2]it/s]
Loading weights [76be5be1b2] from S:\SD ext\models\dl models\epicrealism_pureEvolutionV5.safetensors
Creating model from config: S:\SD_core\stable-diffusion-webui\configs\v1-inference.yaml
Applying attention optimization: xformers... done.
Model loaded in 6.3s (create model: 0.3s, apply weights to model: 5.8s).
100%|███████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.60it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.62it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.59it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.62it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.59it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.61it/s]
Reusing loaded model dl models\epicrealism_pureEvolutionV5.safetensors [76be5be1b2] to load test_0.safetensors [94531057c0]it/s]
Loading weights [94531057c0] from S:\SD ext\models\test_0.safetensors
Creating model from config: S:\SD ext\models\test_0.yaml
Applying attention optimization: xformers... done.
Model loaded in 2.1s (create model: 0.3s, apply weights to model: 1.6s).
100%|███████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.58it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.59it/s]
Initializing dreambooth training...█████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.59it/s]
                                                                                                                                Traceback (most recent call last):                                                                         | 0/1 [00:00<?, ?it/s] 
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\huggingface_hub\utils\_errors.py", line 269, in hf_raise_for_status
    response.raise_for_status()
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\requests\models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/models/test_210.safetensors       

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 730, in start_training
    result = main(class_gen_method=class_gen_method)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1809, in main 
    return inner_loop()
           ^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator       
    return function(batch_size, grad_size, prof, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 543, in inner_loop
    lora_state_dict, network_alphas = LoraLoaderMixin.lora_state_dict(args.lora_model_name)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\diffusers\loaders.py", line 1239, in lora_state_dict
    weight_name = cls._best_guess_weight_name(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\diffusers\loaders.py", line 1288, in _best_guess_weight_name    
    files_in_repo = model_info(pretrained_model_name_or_path_or_dict).siblings
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\huggingface_hub\utils\_validators.py", line 118, in _inner_fn   
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\huggingface_hub\hf_api.py", line 1867, in model_info
    hf_raise_for_status(r)
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\huggingface_hub\utils\_errors.py", line 310, in hf_raise_for_status
    raise RepositoryNotFoundError(message, response) from e
huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-653b60be-26e585f340d3bb9348a8eed7;c0b14eeb-5ce9-4431-977d-10f29919f1b6)

Repository Not Found for url: https://huggingface.co/api/models/test_210.safetensors.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Loading LoRA...:   0%|                                                                                    | 0/1 [00:00<?, ?it/s] 
Duration: 00:00:06
Duration: 00:00:06
Initializing dreambooth training...

===================================BUG REPORT===================================                          | 0/1 [00:00<?, ?it/s]
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('7860" --skip-python-version-check'), WindowsPath('/SD ext/embeddings" --cors-allow-origins="http'), WindowsPath('/127.0.0.1'), WindowsPath('/SD ext/models" --embeddings-dir "S'), WindowsPath(' --api --xformers --ckpt-dir "S')}
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...   
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
WARNING:dreambooth.optimization:Exception importing 8bit AdamW:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Traceback (most recent call last):
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\optimization.py", line 594, in get_optimizer
    from bitsandbytes.optim import AdamW8bit
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\research\__init__.py", line 1, in <module>
    from . import nn
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\research\nn\__init__.py", line 1, in <module>      
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in <module>       
    from bitsandbytes.optim import GlobalOptimManager
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\bitsandbytes\cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
WARNING: Using default optimizer (AdamW from Torch)
                                                                                                                                Init dataset!set:   0%|                                                                                    | 0/1 [00:00<?, ?it/s] 
Preparing Dataset (Without Caching)
Bucket 0 (624, 936, 0) - Instance Images: 27 | Class Images: 0 | Max Examples/batch: 27
Bucket 1 (936, 624, 0) - Instance Images: 15 | Class Images: 0 | Max Examples/batch: 15
Total Buckets 2 - Instance Images: 42 | Class Images: 0 | Max Examples/batch: 42

Total images / batch: 42, total examples: 42█████████████████████████████████████████████████| 42/42 [00:00<00:00, 21001.52it/s] 
                  Initializing bucket counter!
Steps:   0%|                                                                                            | 0/210 [00:00<?, ?it/s]S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\diffusers\models\attention_processor.py:1468: FutureWarning: `LoRAAttnProcessor2_0` is deprecated and will be removed in version 0.26.0. Make sure use AttnProcessor2_0 instead by settingLoRA layers to `del: 0.3s, apply weights to model: 2.4s, move model to device: 1.4s).
Steps:  15%|███████▉                                            | 32/210 [00:52<04:43,  1.59s/it, loss=0.183, lr=5e-5, vram=7.7]Calculating sha256 for S:\SD_core\stable-diffusion-webui\models\Lora\test_210.safetensors: 1d6fa0fba02d80321f488327ad5d8baa0b961d48cba92228337d8a76c4c87ef5
100%|███████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:19<00:00,  1.04it/s] 
Total progress: 100%|███████████████████████████████████████████████████████████████████████████| 20/20 [00:19<00:00,  1.05it/s] 
Steps: 100%|███████████████████████████████████S:\SD_core\stable-diffusion-webui\venv\Lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
  warnings.warn(
 ... (more hidden) ...
 ... (more hidden) ...
 ... (more hidden) ...:   0%|                                                                             | 0/1 [00:00<?, ?it/s] 
                                                                                                                                Model name: testles:   0%|                                                                                 | 0/2 [00:00<?, ?it/s] 
S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\helpers\log_parser.py:324: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
  all_df_loss = all_df_loss.fillna(method="ffill")
Saving S:\SD_core\stable-diffusion-webui\models\dreambooth\test\logging\loss_plot_0.png
Saving S:\SD_core\stable-diffusion-webui\models\dreambooth\test\logging\ram_plot_0.png
Cleanup log parse.
                                                                                                                                Traceback (most recent call last):████████████████████████████▌                                    | 1/2 [00:00<00:00,  3.80it/s] 
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 730, in start_training
    result = main(class_gen_method=class_gen_method)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1809, in main 
    return inner_loop()
           ^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator       
    return function(batch_size, grad_size, prof, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1766, in inner_loop
    check_save(True)
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 976, in check_save
    save_weights(
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1380, in save_weights
    convert_diffusers_to_kohya_lora(lora_save_file, meta, args.lora_weight)
  File "S:\SD_core\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\diff_lora_to_sd_lora.py", line 120, in convert_diffusers_to_kohya_lora
    os.remove(path)
PermissionError: [WinError 5] Zugriff verweigert: 'S:\\SD_core\\stable-diffusion-webui\\models\\Lora\\test_210.safetensors'      
Steps: 100%|█████████████████████████████████████████████████| 210/210 [05:57<00:00,  1.70s/it, loss=0.379, lr=0.0001, vram=7.7] 
Duration: 00:06:03
Generating Samples: 100%|█████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.28it/s]
Duration: 00:06:04

Additional information

No response

wasa4587 commented 8 months ago

Same issue here on windows 10

dreambooth version d8e01e1fd13dd4b6afadbb0382875f905ee5f429

version: v1.6.0

python: 3.10.6

torch: 2.1.0.dev20230712+cu118

xformers: N/A

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary C:\Users\wasa4\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

Shepardnick commented 8 months ago

I've had the same issues, among others. The extension is completely nonfunctional at this time, at least for training SDXL models. I was somehow able to bypass the above issue, but training still wouldn't work. I suspect the case is the same for 2.0 and 1.5.

madrooky commented 8 months ago

I've had the same issues, among others. The extension is completely nonfunctional at this time, at least for training SDXL models. I was somehow able to bypass the above issue, but training still wouldn't work. I suspect the case is the same for 2.0 and 1.5.

Ah yes, i was training with 1.5.

swumagic commented 8 months ago

Bitsandbytes was not supported windows before, but my method can support windows.(yuhuang) 1 open folder J:\StableDiffusion\sdwebui,Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter,cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file(python.exe Folder) here(J:\StableDiffusion\sdwebui\py310)

swumagic commented 8 months ago

OR you are Linux distribution (Ubuntu, MacOS, etc.)system ,AND CUDA Version: 11.X.

Bitsandbytes can support ubuntu.(yuhuang) 1 open folder J:\StableDiffusion\sdwebui,Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter,cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Replace your SD venv directory file(python.exe Folder) here(J:\StableDiffusion\sdwebui\py310)

donlinglok1 commented 8 months ago

Hello all, i got the same issues here (Automatic1111+sd_dreambooth_ext). I think the main problem is

PermissionError: [WinError 5] Zugriff verweigert: 'S:\\SD_core\\stable-diffusion-webui\\models\\Lora\\test_210.safetensors'      

I have try swumagic comment and try reinstall the bitsandbytes-windows. But same PersmissionError happen. Afte i check the python code, i found the issues is in stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\diff_lora_to_sd_lora.py line 120

    os.remove(path)

Basic on the code, my problem is the python cannot remove the file, so i replace the code to this (don't delete file but just rename it): image

    safetensors.torch.save_file(new_model_dict, conv_path, metadata=metadata)
    # Delete the file at path, move the new file to path
    import datetime
    current_datetime=datetime.datetime.now()
    str_date = current_datetime.strftime("%Y-%m-%d_%H%M%S")
    os.rename(path, path+""+str_date+".temp")
    os.rename(conv_path, path)

and now no error popup. This is just a workaround and hope can be fixed gracefully in the feature.

Karlotos commented 7 months ago

I can't get to my PC for logs but I'd like to add that the latest Dreambooth on Auto1111 appears completely broken.

Since fresh install of Auto1111 and dreambooth yesterday:

1) I am unable to create Lora in Automatic1111 with Dreambooth extension, with the same error as above. (PermissionError: [WinError 5] 'C:\... etc)

update: Can confirm the above code fix from 'donlinglok' has worked to resolve this temporarily). The resulting lora still looks burnt though, see problem 2 below.

2) Any model I make with Dreambooth is broken. Any classification images I generate look horrible and 'overcooked' without having done any training, or done anything expect for clicking 'create model' with any 1.5 model I've tried.

3) I cannot make a Dreambooth model at all without having an internet connection. I think it might be trying to check huggingface or something and failing if it can't?

I'm quite new to Dreambooth so could be making an error somewhere but others seem to be having the same issues. Would hugely appreciate some fixes!

I can try to provide more logs or info if useful.

Thanks

donlinglok1 commented 7 months ago

Hi, thanks for the information, actually i am newbie on dreambooth and i have difficulty to tune the training param. Just for your reference now i am using this on train: https://github.com/Akegarasu/lora-scripts

I can't get to my PC for logs but I'd like to add that the latest Dreambooth on Auto1111 appears completely broken.

Since fresh install of Auto1111 and dreambooth yesterday:

  1. I am unable to create Lora in Automatic1111 with Dreambooth extension, with the same error as above. (PermissionError: [WinError 5] 'C:... etc)

update: Can confirm the above code fix from 'donlinglok' has worked to resolve this temporarily). The resulting lora still looks burnt though, see problem 2 below.

  1. Any model I make with Dreambooth is broken. Any classification images I generate look horrible and 'overcooked' without having done any training, or done anything expect for clicking 'create model' with any 1.5 model I've tried.
  2. I cannot make a Dreambooth model at all without having an internet connection. I think it might be trying to check huggingface or something and failing if it can't?

I'm quite new to Dreambooth so could be making an error somewhere but others seem to be having the same issues. Would hugely appreciate some fixes!

I can try to provide more logs or info if useful.

Thanks

PentagramPro commented 7 months ago

I have problem too. When I'm trying to resume training of localy saved LoRa, I get PermissionError. Apparently, it is thinking my Lora file is in Huggingface. I also get occasional WinError 5

github-actions[bot] commented 7 months ago

This issue is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days

thecakeisal1e commented 7 months ago

This is still an issue for me. I've tried reverting to 1.0.14 (ala #1405 ), 3 fresh installs, the bitsandbytes fix in #1371 , and this fix. I've also made sure the folder wasn't read-only, made SD run as an admin, tried a venv and other python envs.

github-actions[bot] commented 6 months ago

This issue is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days

noahjgreer commented 6 months ago

Still having this issue on a fresh install of A1111 and the dreambooth extension. Both are up to date, but the issue still occurs.

Saving weights/samples...: : 0it [00:00, ?it/s]                                                                        Traceback (most recent call last):                                                                | 0/1 [00:00<?, ?it/s]
  File "F:\StableDiffusionProjects\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 735, in start_training
    result = main(class_gen_method=class_gen_method)
  File "F:\StableDiffusionProjects\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1917, in main
    return inner_loop()
  File "F:\StableDiffusionProjects\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator
    return function(batch_size, grad_size, prof, *args, **kwargs)
  File "F:\StableDiffusionProjects\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1874, in inner_loop
    check_save(True)
  File "F:\StableDiffusionProjects\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1027, in check_save
    save_weights(
  File "F:\StableDiffusionProjects\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1433, in save_weights
    convert_diffusers_to_kohya_lora(lora_save_file, meta, args.lora_weight)
  File "F:\StableDiffusionProjects\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\diff_lora_to_sd_lora.py", line 120, in convert_diffusers_to_kohya_lora
    os.remove(path)
PermissionError: [WinError 5] Access is denied: 'F:\\StableDiffusionProjects\\stable-diffusion-webui\\models\\Lora\\lora_Yuley-v1.3.4_I105H_900.safetensors'
PentagramPro commented 5 months ago

Regarding WinError 5

The issue happens in convert_diffusers_to_kohya_lora() when it tries to remove old lora file and replace it with a new one. os.remove() throws an exception. I suspect that what happens is someone occupying the lora file. As a result, os.remove() causes WinError 5. It is unclear, if the file handle is not closed, or there is a race condition with another process. I bypassed it like this:

    # Delete the file at path, move the new file to path
    try:
        os.remove(path)
        os.rename(conv_path, path)
    except:
        pass
github-actions[bot] commented 5 months ago

This issue is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days