Error while trying to train lora with SDXL model

eskaviam commented 1 year ago

[Dataset 0] loading image sizes. 100% 14/14 [00:05<00:00, 2.40it/s] make buckets min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計算されるため、min_bucket_resoとmax_bucket_resoは無視されます number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む） bucket 0: resolution (1024, 1024), count: 392 mean ar error (without repeats): 0.0 noise_offset is set to 0.0357 / noise_offsetが0.0357に設定されました preparing accelerator loading model for process 0/1 load StableDiffusion checkpoint: /content/sdxl_base.safetensors building U-Net loading U-Net from checkpoint U-Net: building text encoders Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 45, in main args.func(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 979, in launch_command simple_launcher(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 628, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command died with <Signals.SIGKILL: 9>.

RejektsAI commented 1 year ago

Same error here, I haven't been able to solve it.

ArthurD1 commented 1 year ago

Hey there, this kind of error can happen when your machine runs out of RAM. Keep an eye on your resources usage

eskaviam commented 1 year ago

Hey there, this kind of error can happen when your machine runs out of RAM. Keep an eye on your resources usage

for my case, after switching to the gui version it fixes it

loboere commented 1 year ago

@eskaviam Can you explain better how you fixed this?

camenduru / kohya_ss-colab

Error while trying to train lora with SDXL model #3