Linaqruf
commented
1 year ago It's not bug, [ "scale_parameter=False", "relative_step=False", "warmup_init=False" ] is arguments used for Adafactor scheduler but you're using AdamW8bit.
Also it's not recommended to use AdamW8bit for SDXL training because Kohya said it doesn't seems to work
==================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 7.5 CUDA SETUP: Detected CUDA version 118 CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so... use 8-bit AdamW optimizer | {'scale_parameter': False, 'relative_step': False, 'warmup_init': False} ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /content/kohya-trainer/sdxl_train_network.py:167 in │
│ │
│ 164 │ args = train_util.read_config_from_file(args, parser) │
│ 165 │ │
│ 166 │ trainer = SdxlNetworkTrainer() │
│ ❱ 167 │ trainer.train(args) │
│ 168 │
│ │
│ /content/kohya-trainer/train_network.py:323 in train │
│ │
│ 320 │ │ │ ) │
│ 321 │ │ │ trainable_params = network.prepare_optimizer_params(args.t │
│ 322 │ │ │
│ ❱ 323 │ │ optimizer_name, optimizer_args, optimizer = train_util.get_opt │
│ 324 │ │ │
│ 325 │ │ # dataloaderを準備する │
│ 326 │ │ # DataLoaderのプロセス数：0はメインプロセスになる │
│ │
│ /content/kohya-trainer/library/train_util.py:2905 in get_optimizer │
│ │
│ 2902 │ │ │ raise ImportError("No bitsand bytes / bitsandbytesがイン │
│ 2903 │ │ print(f"use 8-bit AdamW optimizer | {optimizer_kwargs}") │
│ 2904 │ │ optimizer_class = bnb.optim.AdamW8bit │
│ ❱ 2905 │ │ optimizer = optimizer_class(trainable_params, lr=lr, **optimi │
│ 2906 │ │
│ 2907 │ elif optimizer_type == "SGDNesterov8bit".lower(): │
│ 2908 │ │ try: │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: AdamW8bit.init() got an unexpected keyword argument
'scale_parameter'
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/bin/accelerate:8 in │
│ │
│ 5 from accelerate.commands.accelerate_cli import main │
│ 6 if name == 'main': │
│ 7 │ sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.p │
│ y:45 in main │
│ │
│ 42 │ │ exit(1) │
│ 43 │ │
│ 44 │ # Run │
│ ❱ 45 │ args.func(args) │
│ 46 │
│ 47 │
│ 48 if name == "main": │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:918 in │
│ launch_command │
│ │
│ 915 │ elif defaults is not None and defaults.compute_environment == Comp │
│ 916 │ │ sagemaker_launcher(defaults, args) │
│ 917 │ else: │
│ ❱ 918 │ │ simple_launcher(args) │
│ 919 │
│ 920 │
│ 921 def main(): │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:580 in │
│ simple_launcher │
│ │
│ 577 │ process.wait() │
│ 578 │ if process.returncode != 0: │
│ 579 │ │ if not args.quiet: │
│ ❱ 580 │ │ │ raise subprocess.CalledProcessError(returncode=process.ret │
│ 581 │ │ else: │
│ 582 │ │ │ sys.exit(1) │
│ 583 │
╰──────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/usr/bin/python3', 'sdxl_train_network.py',
'--sample_prompts=/content/LoRA/config/sample_prompt.toml',
'--config_file