[2024-10-09 13:04:03,510] [INFO] [config.py:733:init] Config mesh_device None world_size = 2
[2024-10-09 13:04:13,703] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-10-09 13:04:13,706] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-10-09 13:04:13,706] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-10-09 13:04:13,765] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2024-10-09 13:04:13,765] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2024-10-09 13:04:13,766] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2024-10-09 13:04:13,766] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
rank1: Traceback (most recent call last):
rank1: File "/hy-tmp/sd-scripts/flux_train_network.py", line 519, in
rank1: File "/hy-tmp/sd-scripts/train_network.py", line 595, in train
rank1: ds_model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1303, in prepare
rank1: result = self._prepare_deepspeed(*args)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1779, in _preparedeepspeed
rank1: engine, optimizer, , lr_scheduler = deepspeed.initialize(**kwargs)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize
rank1: engine = DeepSpeedEngine(args=args,
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 313, in initrank1: self._configure_optimizer(optimizer, model_parameters)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1302, in _configure_optimizer
rank1: self.optimizer = self._configure_zero_optimizer(basic_optimizer)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1626, in _configure_zero_optimizer
rank1: optimizer = DeepSpeedZeroOptimizer_Stage3(
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 161, in initrank1: print_rank_0(f"initialized {class.name} with args: {locals()}", force=False)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in reprrank1: list_of_reprs = [repr(item) for item in self]
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in rank1: list_of_reprs = [repr(item) for item in self]
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2546, in reprrank1: extra_repr = self.extra_repr()
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 120, in extra_repr
rank1: return f'in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}'
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in getattrrank1: raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
[2024-10-09 13:04:13,942] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning
[2024-10-09 13:04:13,942] [INFO] [utils.py:782:see_memory_usage] MA 22.17 GB Max_MA 22.17 GB CA 22.18 GB Max_CA 22 GB
[2024-10-09 13:04:13,943] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 88.27 GB, percent = 17.5%
rank0: Traceback (most recent call last):
rank0: File "/hy-tmp/sd-scripts/flux_train_network.py", line 519, in
rank0: File "/hy-tmp/sd-scripts/train_network.py", line 595, in train
rank0: ds_model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1303, in prepare
rank0: result = self._prepare_deepspeed(*args)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1779, in _preparedeepspeed
rank0: engine, optimizer, , lr_scheduler = deepspeed.initialize(**kwargs)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize
rank0: engine = DeepSpeedEngine(args=args,
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 313, in initrank0: self._configure_optimizer(optimizer, model_parameters)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1302, in _configure_optimizer
rank0: self.optimizer = self._configure_zero_optimizer(basic_optimizer)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1626, in _configure_zero_optimizer
rank0: optimizer = DeepSpeedZeroOptimizer_Stage3(
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 161, in initrank0: print_rank_0(f"initialized {class.name} with args: {locals()}", force=False)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in reprrank0: list_of_reprs = [repr(item) for item in self]
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in rank0: list_of_reprs = [repr(item) for item in self]
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in reprrank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2546, in reprrank0: extra_repr = self.extra_repr()
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 120, in extra_repr
rank0: return f'in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}'
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in getattrrank0: raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
W1009 13:04:17.034000 140314240078464 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 6826 closing signal SIGTERM
E1009 13:04:17.600000 140314240078464 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 6825) of binary: /usr/local/miniconda3/envs/ko/bin/python
NeoChen1024
commented
12 hours ago
[2024-10-09 13:04:03,510] [INFO] [config.py:733:init] Config mesh_device None world_size = 2 [2024-10-09 13:04:13,703] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2024-10-09 13:04:13,706] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer [2024-10-09 13:04:13,706] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer [2024-10-09 13:04:13,765] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam [2024-10-09 13:04:13,765] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'> [2024-10-09 13:04:13,766] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False [2024-10-09 13:04:13,766] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer rank1: Traceback (most recent call last): rank1: File "/hy-tmp/sd-scripts/flux_train_network.py", line 519, in
rank1: File "/hy-tmp/sd-scripts/train_network.py", line 595, in train rank1: ds_model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1303, in prepare rank1: result = self._prepare_deepspeed(*args)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1779, in _preparedeepspeed rank1: engine, optimizer, , lr_scheduler = deepspeed.initialize(**kwargs)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize rank1: engine = DeepSpeedEngine(args=args,
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 313, in init rank1: self._configure_optimizer(optimizer, model_parameters) rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1302, in _configure_optimizer rank1: self.optimizer = self._configure_zero_optimizer(basic_optimizer)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1626, in _configure_zero_optimizer rank1: optimizer = DeepSpeedZeroOptimizer_Stage3(
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 161, in init rank1: print_rank_0(f"initialized {class.name} with args: {locals()}", force=False)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in repr rank1: list_of_reprs = [repr(item) for item in self]
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in
rank1: list_of_reprs = [repr(item) for item in self]
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank1: mod_str = repr(module)
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2546, in repr rank1: extra_repr = self.extra_repr()
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 120, in extra_repr rank1: return f'in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}'
rank1: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in getattr rank1: raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
[2024-10-09 13:04:13,942] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning [2024-10-09 13:04:13,942] [INFO] [utils.py:782:see_memory_usage] MA 22.17 GB Max_MA 22.17 GB CA 22.18 GB Max_CA 22 GB [2024-10-09 13:04:13,943] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 88.27 GB, percent = 17.5% rank0: Traceback (most recent call last): rank0: File "/hy-tmp/sd-scripts/flux_train_network.py", line 519, in
rank0: File "/hy-tmp/sd-scripts/train_network.py", line 595, in train rank0: ds_model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1303, in prepare rank0: result = self._prepare_deepspeed(*args)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1779, in _preparedeepspeed rank0: engine, optimizer, , lr_scheduler = deepspeed.initialize(**kwargs)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/init.py", line 193, in initialize rank0: engine = DeepSpeedEngine(args=args,
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 313, in init rank0: self._configure_optimizer(optimizer, model_parameters) rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1302, in _configure_optimizer rank0: self.optimizer = self._configure_zero_optimizer(basic_optimizer)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1626, in _configure_zero_optimizer rank0: optimizer = DeepSpeedZeroOptimizer_Stage3(
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 161, in init rank0: print_rank_0(f"initialized {class.name} with args: {locals()}", force=False)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in repr rank0: list_of_reprs = [repr(item) for item in self]
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in
rank0: list_of_reprs = [repr(item) for item in self]
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in repr rank0: mod_str = repr(module)
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2546, in repr rank0: extra_repr = self.extra_repr()
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 120, in extra_repr rank0: return f'in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}'
rank0: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in getattr rank0: raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
W1009 13:04:17.034000 140314240078464 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 6826 closing signal SIGTERM E1009 13:04:17.600000 140314240078464 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 6825) of binary: /usr/local/miniconda3/envs/ko/bin/python
here is my command:
accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 4 flux_train_network.py --pretrained_model_name_or_path /hy-tmp/flux/flux1-dev.safetensors --clip_l /hy-tmp/flux/clip_l.safetensors --t5xxl /hy-tmp/flux/t5xxl_fp16.safetensors --ae /hy-tmp/flux/ae.safetensors --cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers --max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 --network_module lycoris.kohya --network_dim 32 --network_alpha 16 --network_args "conv_dim=32" "conv_alpha=16" "algo=full" "bypass_mode=false" --optimizer_type "AdamW" --learning_rate 1e-4 --full_bf16 --network_train_unet_only --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --max_train_epochs 3 --save_every_n_epochs 1 --dataset_config anomaly_bearing.toml --output_dir /hy-tmp/out --output_name anomaly_bearing_lyco_mk1 --timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1.0 --loss_type l2 --lr_scheduler constant_with_warmup --max_grad_norm 0.0 --xformers --deepspeed --zero_stage=3 --offload_optimizer_device="cpu" --offload_param_device="cpu"