error while training lycoris for flux.1

AUTOMATIC2222 commented 1 month ago

[2024-10-09 13:04:03,510] [INFO] [config.py:733:init] Config mesh_device None world_size = 2 [2024-10-09 13:04:13,703] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2024-10-09 13:04:13,706] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer [2024-10-09 13:04:13,706] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer [2024-10-09 13:04:13,765] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam [2024-10-09 13:04:13,765] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'> [2024-10-09 13:04:13,766] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False [2024-10-09 13:04:13,766] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer rank1: Traceback (most recent call last): rank1: File "/hy-tmp/sd-scripts/flux_train_network.py", line 519, in

rank1: File "/hy-tmp/sd-scripts/train_network.py", line 595, in train rank1: ds_model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1303, in prepare rank1: result = self._prepare_deepspeed(*args)

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1779, in _preparedeepspeed rank1: engine, optimizer, , lr_scheduler = deepspeed.initialize(**kwargs)

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize rank1: engine = DeepSpeedEngine(args=args,

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 313, in init rank1: self._configure_optimizer(optimizer, model_parameters) rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1302, in _configure_optimizer rank1: self.optimizer = self._configure_zero_optimizer(basic_optimizer)

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1626, in _configure_zero_optimizer rank1: optimizer = DeepSpeedZeroOptimizer_Stage3(

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 161, in init rank1: print_rank_0(f"initialized {class.name} with args: {locals()}", force=False)

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank1: mod_str = repr(module)

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in repr rank1: list_of_reprs = [repr(item) for item in self]

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in rank1: list_of_reprs = [repr(item) for item in self]

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank1: mod_str = repr(module)

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2546, in repr rank1: extra_repr = self.extra_repr()

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 120, in extra_repr rank1: return f'in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}'

rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in getattr rank1: raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")

[2024-10-09 13:04:13,942] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning [2024-10-09 13:04:13,942] [INFO] [utils.py:782:see_memory_usage] MA 22.17 GB Max_MA 22.17 GB CA 22.18 GB Max_CA 22 GB [2024-10-09 13:04:13,943] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 88.27 GB, percent = 17.5% rank0: Traceback (most recent call last): rank0: File "/hy-tmp/sd-scripts/flux_train_network.py", line 519, in

rank0: File "/hy-tmp/sd-scripts/train_network.py", line 595, in train rank0: ds_model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1303, in prepare rank0: result = self._prepare_deepspeed(*args)

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1779, in _preparedeepspeed rank0: engine, optimizer, , lr_scheduler = deepspeed.initialize(**kwargs)

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize rank0: engine = DeepSpeedEngine(args=args,

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 313, in init rank0: self._configure_optimizer(optimizer, model_parameters) rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1302, in _configure_optimizer rank0: self.optimizer = self._configure_zero_optimizer(basic_optimizer)

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1626, in _configure_zero_optimizer rank0: optimizer = DeepSpeedZeroOptimizer_Stage3(

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 161, in init rank0: print_rank_0(f"initialized {class.name} with args: {locals()}", force=False)

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank0: mod_str = repr(module)

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in repr rank0: list_of_reprs = [repr(item) for item in self]

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in rank0: list_of_reprs = [repr(item) for item in self]

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank0: mod_str = repr(module)

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2546, in repr rank0: extra_repr = self.extra_repr()

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 120, in extra_repr rank0: return f'in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}'

rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in getattr rank0: raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")

W1009 13:04:17.034000 140314240078464 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 6826 closing signal SIGTERM E1009 13:04:17.600000 140314240078464 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 6825) of binary: /usr/local/miniconda3/envs/ko/bin/python

here is my command:

accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 4 flux_train_network.py --pretrained_model_name_or_path /hy-tmp/flux/flux1-dev.safetensors --clip_l /hy-tmp/flux/clip_l.safetensors --t5xxl /hy-tmp/flux/t5xxl_fp16.safetensors --ae /hy-tmp/flux/ae.safetensors --cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers --max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 --network_module lycoris.kohya --network_dim 32 --network_alpha 16 --network_args "conv_dim=32" "conv_alpha=16" "algo=full" "bypass_mode=false" --optimizer_type "AdamW" --learning_rate 1e-4 --full_bf16 --network_train_unet_only --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --max_train_epochs 3 --save_every_n_epochs 1 --dataset_config anomaly_bearing.toml --output_dir /hy-tmp/out --output_name anomaly_bearing_lyco_mk1 --timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1.0 --loss_type l2 --lr_scheduler constant_with_warmup --max_grad_norm 0.0 --xformers --deepspeed --zero_stage=3 --offload_optimizer_device="cpu" --offload_param_device="cpu"

NeoChen1024 commented 1 month ago

Same problem here.

AUTOMATIC2222 commented 1 month ago

@kohya-ss hi,do you have any suggestion?

heinrichI commented 1 day ago

I also have error on attempt train LoHA: AttributeError: 'LycorisNetwork' object has no attribute 'train_t5xxl' :

_python.exe 'sd-scripts/flux_train_network.py', '--pretrained_model_name_or_path', 'flux1-dev-fp8.safetensors', '--clip_l', 'clip_l.safetensors', '--t5xxl', 't5xxl_fp16.safetensors', '--ae', 'ae.sft', '--cache_latents_to_disk', '--cache_text_encoder_outputs', '--cache_text_encoder_outputs_to_disk', '--save_model_as', 'safetensors', '--sdpa', '--persistent_data_loader_workers', '--max_data_loader_n_workers', '2', '--seed', '42', '--gradient_checkpointing', '--mixed_precision', 'bf16', '--save_precision', 'bf16', '--network_module', 'lycoris.kohya', '--network_dim', '32', '--network_alpha', '1', '--network_args', 'conv_dim=32', 'conv_alpha=1', 'algo=loha', '--optimizer_type', 'adafactor', '--optimizer_args', 'relative_step=False', 'scale_parameter=False', 'warmup_init=False', '--lr_scheduler', 'constant_with_warmup', '--max_grad_norm', '0.0', '--learning_rate', '5e-4', '--network_train_unet_only', '--fp8_base_unet', '--highvram', '--max_train_epochs', '100', '--save_every_n_epochs', '1', '--dataset_config', 'datasetMeMaskReg.toml', '--output_dir', 'outputs', '--output_name', 'me5e4AdaFactorMaskReg', '--timestep_sampling', 'shift', '--discrete_flow_shift', '3.1582', '--model_prediction_type', 'raw', '--guidance_scale', '1', '--loss_type', 'l2', '--sample_every_n_epochs', '1', '--sample_prompts', 'MePromtSmall.txt', '--sample_sampler', 'euler_a', '--save_state', '--save_state_on_train_end', '--apply_t5_attn_mask', '--masked_loss', '--logging_dir', './logs', '--log_with', 'tensorboard', '--log_trackername', 'me5e4AdaFactorMaskReg'

kohya-ss / sd-scripts

error while training lycoris for flux.1 #1683