TeKett
commented
11 months ago Can confirm, exact same issue. I downloaded v22.2.1 as a completely new setup today, since i wanted to test Lion, which was not working (due to older bitsandbytes version) on my current setup. Cant confirm nor do i have the time too if its just the gui or not since i dont use the non-gui version, so maybe we should crosspost this issue to the main repo (https://github.com/kohya-ss/sd-scripts).
07:29:40-427896 INFO Start Finetuning...
07:29:40-522670 INFO image_num = 16680
07:29:40-523640 INFO repeats = 16680
07:29:40-525635 INFO max_train_steps = 333600
07:29:40-526661 INFO lr_warmup_steps = 0
07:29:40-527629 INFO Saving training config to C:/Users/user/stable-diffusion
/Train/checkpoint/model\checkpoint_20231117-072940.json...
07:29:40-529624 INFO accelerate launch --num_cpu_threads_per_process=1 "./fine_tune.py" --train_text_encoder
--learning_rate_te="1e-05"
--pretrained_model_name_or_path="C:/Users/user/stable-diffusion-webui/models/Stable-diffusion
/#New folder/checkpoint-000018.safetensors" --in_json="C:/Users/user/stable-diffusion
/Train/checkpoint/config/meta_lat.json" --train_data_dir="C:/Users/user/stable-diffusion
/Train/checkpoint/img/checkpoint" --output_dir="C:/Users/user/stable-diffusion
/Train/checkpoint/model" --logging_dir="C:/Users/user/stable-diffusion /Train/checkpoint/log"
--dataset_repeats=1 --enable_bucket --resolution="512,768" --min_bucket_reso=256
--max_bucket_reso=1024 --save_model_as=safetensors --output_name="checkpoint"
--max_token_length=225 --learning_rate="1e-05" --lr_scheduler="cosine" --train_batch_size="1"
--max_train_steps="333600" --save_every_n_epochs="1" --mixed_precision="bf16"
--save_precision="bf16" --caption_extension=".txt" --cache_latents --cache_latents_to_disk
--optimizer_type="Lion8bit" --max_data_loader_n_workers="0" --max_token_length=225
--bucket_reso_steps=8 --min_timestep=500 --max_timestep=650 --xformers --noise_offset=0.0
prepare tokenizer
update token length: 225
loading existing metadata: C:/Users/user/stable-diffusion /Train/checkpoint/config/meta_lat.json
using bucket info in metadata / メタデータ内のbucket情報を使います
[Dataset 0]
batch_size: 1
resolution: (512, 768)
enable_bucket: True
min_bucket_reso: None
max_bucket_reso: None
bucket_reso_steps: None
bucket_no_upscale: None
[Subset 0 of Dataset 0]
image_dir: "C:/Users/user/stable-diffusion /Train/checkpoint/img/1_name"
image_count: 16680
num_repeats: 1
shuffle_caption: False
keep_tokens: 0
caption_dropout_rate: 0.0
caption_dropout_every_n_epoches: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: False
token_warmup_min: 1,
token_warmup_step: 0,
metadata_file: C:/Users/user/stable-diffusion /Train/checkpoint/config/meta_lat.json
[Dataset 0]
loading image sizes.
100%|███████████████████████████████████████████████████████████████████████| 16680/16680 [00:00<00:00, 4156674.63it/s]
make buckets
number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）
bucket 0: resolution (256, 1024), count: 4
bucket 1: resolution (320, 1024), count: 43
bucket 2: resolution (384, 896), count: 205
bucket 3: resolution (384, 960), count: 56
bucket 4: resolution (384, 1024), count: 31
bucket 5: resolution (448, 832), count: 1649
bucket 6: resolution (512, 704), count: 5403
bucket 7: resolution (512, 768), count: 3744
bucket 8: resolution (576, 576), count: 1235
bucket 9: resolution (576, 640), count: 1628
bucket 10: resolution (640, 576), count: 684
bucket 11: resolution (704, 512), count: 844
bucket 12: resolution (768, 512), count: 391
bucket 13: resolution (832, 448), count: 760
bucket 14: resolution (896, 384), count: 3
mean ar error (without repeats): 0.0
prepare accelerator
loading model for process 0/1
load StableDiffusion checkpoint: C:/Users/user/stable-diffusion-webui/models/Stable-diffusion/#New folder/checkpoint-000018.safetensors
UNet2DConditionModel: 64, 8, 768, False, False
loading u-net: <All keys matched successfully>
loading vae: <All keys matched successfully>
loading text encoder: <All keys matched successfully>
Disable Diffusers' xformers
Enable xformers for U-Net
[Dataset 0]
caching latents.
checking cache validity...
100%|███████████████████████████████████████████████████████████████████████| 16680/16680 [00:00<00:00, 2388480.79it/s]
caching latents...
0it [00:00, ?it/s]
enable text encoder training
prepare optimizer, data loader etc.
Traceback (most recent call last):
File "C:\Users\user\Kohya 2\library\train_util.py", line 3433, in get_optimizer
import bitsandbytes as bnb
File "C:\Users\user\Kohya 2\venv\lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
from . import cuda_setup, utils, research
File "C:\Users\user\Kohya 2\venv\lib\site-packages\bitsandbytes\research\__init__.py", line 1, in <module>
from . import nn
File "C:\Users\user\Kohya 2\venv\lib\site-packages\bitsandbytes\research\nn\__init__.py", line 1, in <module>
from .modules import LinearFP8Mixed, LinearFP8Global
File "C:\Users\user\Kohya 2\venv\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in <module>
from bitsandbytes.optim import GlobalOptimManager
File "C:\Users\user\Kohya 2\venv\lib\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module>
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "C:\Users\user\Kohya 2\venv\lib\site-packages\bitsandbytes\cextension.py", line 5, in <module>
from .cuda_setup.main import evaluate_cuda_setup
File "C:\Users\user\Kohya 2\venv\lib\site-packages\bitsandbytes\cuda_setup\main.py", line 21, in <module>
from .paths import determine_cuda_runtime_lib_path
ModuleNotFoundError: No module named 'bitsandbytes.cuda_setup.paths'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\Kohya 2\fine_tune.py", line 499, in <module>
train(args)
File "C:\Users\user\Kohya 2\fine_tune.py", line 212, in train
_, _, optimizer = train_util.get_optimizer(args, trainable_params=trainable_params)
File "C:\Users\user\Kohya 2\library\train_util.py", line 3435, in get_optimizer
raise ImportError("No bitsandbytes / bitsandbytesがインストールされていないようです")
ImportError: No bitsandbytes / bitsandbytesがインストールされていないようです
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\user\Kohya 2\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
File "C:\Users\user\Kohya 2\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "C:\Users\user\Kohya 2\venv\lib\site-packages\accelerate\commands\launch.py", line 986, in launch_command
simple_launcher(args)
File "C:\Users\user\Kohya 2\venv\lib\site-packages\accelerate\commands\launch.py", line 628, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['C:\\Users\\user\\Kohya 2\\venv\\Scripts\\python.exe', './fine_tune.py', '--train_text_encoder', '--learning_rate_te=1e-05', '--pretrained_model_name_or_path=C:/Users/user/stable-diffusion-webui/models/Stable-diffusion/#New folder/checkpoint-000018.safetensors', '--in_json=C:/Users/user/stable-diffusion /Train/checkpoint/config/meta_lat.json', '--train_data_dir=C:/Users/user/stable-diffusion /Train/checkpoint/img/1_name', '--output_dir=C:/Users/user/stable-diffusion /Train/checkpoint/model', '--logging_dir=C:/Users/user/stable-diffusion /Train/checkpoint/log', '--dataset_repeats=1', '--enable_bucket', '--resolution=512,768', '--min_bucket_reso=256', '--max_bucket_reso=1024', '--save_model_as=safetensors', '--output_name=checkpoint', '--max_token_length=225', '--learning_rate=1e-05', '--lr_scheduler=cosine', '--train_batch_size=1', '--max_train_steps=333600', '--save_every_n_epochs=1', '--mixed_precision=bf16', '--save_precision=bf16', '--caption_extension=.txt', '--cache_latents', '--cache_latents_to_disk', '--optimizer_type=Lion8bit', '--max_data_loader_n_workers=0', '--max_token_length=225', '--bucket_reso_steps=8', '--min_timestep=500', '--max_timestep=650', '--xformers', '--noise_offset=0.0']' returned non-zero exit status 1.
TheAnay
Aamir3d
Wreggor
This has been posted further down the chain but has no traction or responses.
Have been getting a lot of this attempting to train.
I can get it to run with adafactor with a 0 LR warmup, but can't really get it going in any other configuration