bmaltais / kohya_ss

Apache License 2.0
9.64k stars 1.24k forks source link

returned non-zero exit status 1. #1334

Closed db874hjiXc3bT01b5 closed 9 months ago

db874hjiXc3bT01b5 commented 1 year ago

I reinstalled Kohya on a new PC and run into this every time I attempt to train a LoRA.

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ D:\webui\kohya\kohya_ss\train_network.py:991 in <module> │ │ │ │ 988 │ args = train_util.read_config_from_file(args, parser) │ │ 989 │ │ │ 990 │ trainer = NetworkTrainer() │ │ ❱ 991 │ trainer.train(args) │ │ 992 │ │ │ │ D:\webui\kohya\kohya_ss\train_network.py:324 in train │ │ │ │ 321 │ │ │ ) │ │ 322 │ │ │ trainable_params = network.prepare_optimizer_params(args.text_encoder_lr, ar │ │ 323 │ │ │ │ ❱ 324 │ │ optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, train │ │ 325 │ │ │ │ 326 │ │ # dataloaderを準備する │ │ 327 │ │ # DataLoaderのプロセス数:0はメインプロセスになる │ │ │ │ D:\webui\kohya\kohya_ss\library\train_util.py:3204 in get_optimizer │ │ │ │ 3201 │ │ │ 3202 │ elif optimizer_type.endswith("8bit".lower()): │ │ 3203 │ │ try: │ │ ❱ 3204 │ │ │ import bitsandbytes as bnb │ │ 3205 │ │ except ImportError: │ │ 3206 │ │ │ raise ImportError("No bitsandbytes / bitsandbytesがインストールされていない │ │ 3207 │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py:6 in <module> │ │ │ │ 3 # This source code is licensed under the MIT license found in the │ │ 4 # LICENSE file in the root directory of this source tree. │ │ 5 │ │ ❱ 6 from .autograd._functions import ( │ │ 7 │ MatmulLtState, │ │ 8 │ bmm_cublas, │ │ 9 │ matmul, │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\autograd\_functions.py:5 in <module> │ │ │ │ 2 import warnings │ │ 3 │ │ 4 import torch │ │ ❱ 5 import bitsandbytes.functional as F │ │ 6 │ │ 7 from dataclasses import dataclass │ │ 8 from functools import reduce # Required in Python 3 │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\functional.py:13 in <module> │ │ │ │ 10 from typing import Tuple │ │ 11 from torch import Tensor │ │ 12 │ │ ❱ 13 from .cextension import COMPILED_WITH_CUDA, lib │ │ 14 from functools import reduce # Required in Python 3 │ │ 15 │ │ 16 # math.prod not compatible with python < 3.8 │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py:41 in <module> │ │ │ │ 38 │ │ return cls._instance │ │ 39 │ │ 40 │ │ ❱ 41 lib = CUDALibrary_Singleton.get_instance().lib │ │ 42 try: │ │ 43 │ lib.cadam32bit_g32 │ │ 44 │ lib.get_context.restype = ct.c_void_p │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py:37 in get_instance │ │ │ │ 34 │ def get_instance(cls): │ │ 35 │ │ if cls._instance is None: │ │ 36 │ │ │ cls._instance = cls.__new__(cls) │ │ ❱ 37 │ │ │ cls._instance.initialize() │ │ 38 │ │ return cls._instance │ │ 39 │ │ 40 │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py:31 in initialize │ │ │ │ 28 │ │ │ self.lib = ct.cdll.LoadLibrary(binary_path) │ │ 29 │ │ else: │ │ 30 │ │ │ print(f"CUDA SETUP: Loading binary {binary_path}...") │ │ ❱ 31 │ │ │ self.lib = ct.cdll.LoadLibrary(binary_path) │ │ 32 │ │ │ 33 │ @classmethod │ │ 34 │ def get_instance(cls): │ │ │ │ C:\Users\My PC\AppData\Local\Programs\Python\Python310\lib\ctypes\__init__.py:452 in │ │ LoadLibrary │ │ │ │ 449 │ │ return getattr(self, name) │ │ 450 │ │ │ 451 │ def LoadLibrary(self, name): │ │ ❱ 452 │ │ return self._dlltype(name) │ │ 453 │ │ │ 454 │ __class_getitem__ = classmethod(_types.GenericAlias) │ │ 455 │ │ │ │ C:\Users\My PC\AppData\Local\Programs\Python\Python310\lib\ctypes\__init__.py:364 in __init__ │ │ │ │ 361 │ │ │ else: │ │ 362 │ │ │ │ import nt │ │ 363 │ │ │ │ mode = nt._LOAD_LIBRARY_SEARCH_DEFAULT_DIRS │ │ ❱ 364 │ │ │ │ if '/' in name or '\\' in name: │ │ 365 │ │ │ │ │ self._name = nt._getfullpathname(self._name) │ │ 366 │ │ │ │ │ mode |= nt._LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR │ │ 367 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ TypeError: argument of type 'WindowsPath' is not iterable ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ C:\Users\My PC\AppData\Local\Programs\Python\Python310\lib\runpy.py:196 in _run_module_as_main │ │ │ │ 193 │ main_globals = sys.modules["__main__"].__dict__ │ │ 194 │ if alter_argv: │ │ 195 │ │ sys.argv[0] = mod_spec.origin │ │ ❱ 196 │ return _run_code(code, main_globals, None, │ │ 197 │ │ │ │ │ "__main__", mod_spec) │ │ 198 │ │ 199 def run_module(mod_name, init_globals=None, │ │ │ │ C:\Users\My PC\AppData\Local\Programs\Python\Python310\lib\runpy.py:86 in _run_code │ │ │ │ 83 │ │ │ │ │ __loader__ = loader, │ │ 84 │ │ │ │ │ __package__ = pkg_name, │ │ 85 │ │ │ │ │ __spec__ = mod_spec) │ │ ❱ 86 │ exec(code, run_globals) │ │ 87 │ return run_globals │ │ 88 │ │ 89 def _run_module_code(code, init_globals=None, │ │ │ │ in <module>:7 │ │ │ │ 4 from accelerate.commands.accelerate_cli import main │ │ 5 if __name__ == '__main__': │ │ 6 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │ │ ❱ 7 │ sys.exit(main()) │ │ 8 │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py:45 in main │ │ │ │ 42 │ │ exit(1) │ │ 43 │ │ │ 44 │ # Run │ │ ❱ 45 │ args.func(args) │ │ 46 │ │ 47 │ │ 48 if __name__ == "__main__": │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py:918 in │ │ launch_command │ │ │ │ 915 │ elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA │ │ 916 │ │ sagemaker_launcher(defaults, args) │ │ 917 │ else: │ │ ❱ 918 │ │ simple_launcher(args) │ │ 919 │ │ 920 │ │ 921 def main(): │ │ │ │ D:\webui\kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py:580 in │ │ simple_launcher │ │ │ │ 577 │ process.wait() │ │ 578 │ if process.returncode != 0: │ │ 579 │ │ if not args.quiet: │ │ ❱ 580 │ │ │ raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) │ │ 581 │ │ else: │ │ 582 │ │ │ sys.exit(1) │ │ 583 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ CalledProcessError: Command '['D:\\webui\\kohya\\kohya_ss\\venv\\Scripts\\python.exe', './train_network.py', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--pretrained_model_name_or_path=D:\\webui\\stable-diffusion-webui\\models\\Stable-diffusion\\hassakuHentaiModel_v11.saf etensors', '--train_data_dir=D:/webui/dataset images/WaterGoddessAqua/images', '--resolution=768,768', '--output_dir=D:/webui/dataset images/WaterGoddessAqua/model', '--logging_dir=D:/webui/dataset images/WaterGoddessAqua/log', '--network_alpha=180', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=5e-05', '--unet_lr=0.0001', '--network_dim=180', '--output_name=WaterGoddessAqua_Exotica', '--lr_scheduler_num_cycles=1', '--no_half_vae', '--learning_rate=0.0001', '--lr_scheduler=constant', '--train_batch_size=4', '--max_train_steps=503', '--save_every_n_epochs=1', '--mixed_precision=fp16', '--save_precision=fp16', '--cache_latents', '--optimizer_type=AdamW8bit', '--max_data_loader_n_workers=0', '--clip_skip=2', '--bucket_reso_steps=64', '--xformers', '--bucket_no_upscale', '--noise_offset=0.0']' returned non-zero exit status 1.

juanigsrz commented 1 year ago

Try putting the directory at the root of the disk

diegomagikal commented 1 year ago

Try putting the directory at the root of the disk

It worked for me, thank you.

Gui205 commented 1 year ago

I have the same exact problem, i have all the file on the root of my C disk => C:\kohya_ss

cwinebrenner commented 1 year ago

if your using the 2.x model try using the 1.5 model, fixed it for me

Gui205 commented 1 year ago

if your using the 2.x model try using the 1.5 model, fixed it for me

Hi ! Are you on AMD or Nvidia ?

cwinebrenner commented 1 year ago

Nvidia rtx

On Tue, Aug 15, 2023, 8:04 PM Gui205 @.***> wrote:

if your using the 2.x model try using the 1.5 model, fixed it for me

Hi ! Are you on AMD or Nvidia ?

— Reply to this email directly, view it on GitHub https://github.com/bmaltais/kohya_ss/issues/1334#issuecomment-1679892795, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBPRYDEX46WD4NQBZGO4LHLXVQ2FTANCNFSM6AAAAAA3FUZGLA . You are receiving this because you commented.Message ID: @.***>

ialabmex commented 10 months ago

I found this solution that worked perfectly for me: changing the optimizer to the non-8-bit version.

Changing AdamW8bit, to AdamW in the parameters tab and the training worked without problems.

ref: https://stackoverflow.com/questions/77506481/error-when-training-lora-model-using-kohya-ss