search

huggingface / accelerate

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support
https://huggingface.co/docs/accelerate
Apache License 2.0
7.98k stars 974 forks source link

Communication problems with deepspeed zero3 #3240

Closed Reginald-L closed 1 week ago

Reginald-L commented 1 week ago

Hi， I am using accelerate and deepspeed's zero3 to fine tune a flux model, I have run through the process of deepspeed zero3 on a single GPU.

But when I make the number of GPU's > 1, there is alway the timeout issue. so I guess it's a communication problem between multiple GPUs

I try some solutions:

Here is related code and output:

guidance_vec = torch.full((img.shape[0],), guidance, device=img.device, dtype=img.dtype)
model.prepare_block_swap_before_forward()

for t_curr, t_prev in zip(tqdm(timesteps[:-1]), timesteps[1:]):
    t_vec = torch.full((img.shape[0],), t_curr, dtype=img.dtype, device=img.device) 
    pred = model(
        img=img.to(model.dtype).to(model.device),
        img_ids=img_ids.to(model.dtype).to(model.device),
        txt=txt.to(model.dtype).to(model.device),
        txt_ids=txt_ids.to(model.dtype).to(model.device),
        y=vec.to(model.dtype).to(model.device),
        timesteps=t_vec,
        guidance=guidance_vec,
        txt_attention_mask=t5_attn_mask.to(model.dtype).to(model.device) if t5_attn_mask is not None else t5_attn_mask,
    )
    img = img + (t_prev - t_curr) * pred

model.prepare_block_swap_before_forward()
return img

image image Here is my config: image

here is config related distribution:

================== EXECUTE prepare deepspeed model ==================
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.001000, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1
[2024-11-16 18:20:59,089] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.001000, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1
[2024-11-16 18:20:59,303] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.15.4, git-hash=unknown, git-branch=unknown
[2024-11-16 18:20:59,304] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[2024-11-16 18:20:59,464] [INFO] [utils.py:781:see_memory_usage] DeepSpeed Engine: After args sanity test
[2024-11-16 18:20:59,464] [INFO] [utils.py:782:see_memory_usage] MA 9.11 GB         Max_MA 9.35 GB         CA 9.37 GB         Max_CA 9 GB
[2024-11-16 18:20:59,464] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 35.59 GB, percent = 2.4%
[2024-11-16 18:20:59,593] [INFO] [utils.py:781:see_memory_usage] DeepSpeed Engine: Before configure distributed model
[2024-11-16 18:20:59,593] [INFO] [utils.py:782:see_memory_usage] MA 9.11 GB         Max_MA 9.11 GB         CA 9.37 GB         Max_CA 9 GB
[2024-11-16 18:20:59,593] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 35.59 GB, percent = 2.4%
Tencent-L20:1157794:1157794 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
Tencent-L20:1157794:1157794 [0] NCCL INFO Bootstrap : Using eth0:172.21.56.11<0>
Tencent-L20:1157794:1157794 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
Tencent-L20:1157794:1157794 [0] NCCL INFO cudaDriverVersion 12020
NCCL version 2.20.5+cuda12.4
Tencent-L20:1157795:1157795 [1] NCCL INFO cudaDriverVersion 12020
Tencent-L20:1157795:1157795 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
Tencent-L20:1157795:1157795 [1] NCCL INFO Bootstrap : Using eth0:172.21.56.11<0>
Tencent-L20:1157795:1157795 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
Tencent-L20:1157794:1158348 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
Tencent-L20:1157794:1158348 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
Tencent-L20:1157794:1158348 [0] NCCL INFO NET/Socket : Using [0]eth0:172.21.56.11<0>
Tencent-L20:1157794:1158348 [0] NCCL INFO Using non-device net plugin version 0
Tencent-L20:1157794:1158348 [0] NCCL INFO Using network Socket
Tencent-L20:1157795:1158349 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
Tencent-L20:1157795:1158349 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
Tencent-L20:1157795:1158349 [1] NCCL INFO NET/Socket : Using [0]eth0:172.21.56.11<0>
Tencent-L20:1157795:1158349 [1] NCCL INFO Using non-device net plugin version 0
Tencent-L20:1157795:1158349 [1] NCCL INFO Using network Socket
Tencent-L20:1157795:1158349 [1] NCCL INFO comm 0x10cca2a0 rank 1 nranks 2 cudaDev 1 nvmlDev 4 busId 63000 commId 0xaa3395e05bedbf02 - Init START
Tencent-L20:1157794:1158348 [0] NCCL INFO comm 0x11951a20 rank 0 nranks 2 cudaDev 0 nvmlDev 3 busId 43000 commId 0xaa3395e05bedbf02 - Init START
Tencent-L20:1157795:1158349 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
Tencent-L20:1157794:1158348 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
Tencent-L20:1157795:1158349 [1] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,00000000,00000000,00000000,00000000,00000000,00000000
Tencent-L20:1157794:1158348 [0] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
Tencent-L20:1157795:1158349 [1] NCCL INFO comm 0x10cca2a0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
Tencent-L20:1157795:1158349 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
Tencent-L20:1157795:1158349 [1] NCCL INFO P2P Chunksize set to 131072
Tencent-L20:1157794:1158348 [0] NCCL INFO comm 0x11951a20 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
Tencent-L20:1157794:1158348 [0] NCCL INFO Channel 00/02 :    0   1
Tencent-L20:1157794:1158348 [0] NCCL INFO Channel 01/02 :    0   1
Tencent-L20:1157794:1158348 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
Tencent-L20:1157794:1158348 [0] NCCL INFO P2P Chunksize set to 131072
Tencent-L20:1157795:1158349 [1] NCCL INFO Channel 00 : 1[4] -> 0[3] via SHM/direct/direct
Tencent-L20:1157795:1158349 [1] NCCL INFO Channel 01 : 1[4] -> 0[3] via SHM/direct/direct
Tencent-L20:1157794:1158348 [0] NCCL INFO Channel 00 : 0[3] -> 1[4] via SHM/direct/direct
Tencent-L20:1157794:1158348 [0] NCCL INFO Channel 01 : 0[3] -> 1[4] via SHM/direct/direct
Tencent-L20:1157794:1158348 [0] NCCL INFO Connected all rings
Tencent-L20:1157794:1158348 [0] NCCL INFO Connected all trees
Tencent-L20:1157795:1158349 [1] NCCL INFO Connected all rings
Tencent-L20:1157795:1158349 [1] NCCL INFO Connected all trees
Tencent-L20:1157795:1158349 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
Tencent-L20:1157795:1158349 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
Tencent-L20:1157794:1158348 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
Tencent-L20:1157794:1158348 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
Tencent-L20:1157795:1158349 [1] NCCL INFO comm 0x10cca2a0 rank 1 nranks 2 cudaDev 1 nvmlDev 4 busId 63000 commId 0xaa3395e05bedbf02 - Init COMPLETE
Tencent-L20:1157794:1158348 [0] NCCL INFO comm 0x11951a20 rank 0 nranks 2 cudaDev 0 nvmlDev 3 busId 43000 commId 0xaa3395e05bedbf02 - Init COMPLETE
[2024-11-16 18:21:04,588] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-11-16 18:21:04,590] [INFO] [logging.py:128:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-11-16 18:21:04,590] [INFO] [logging.py:128:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-11-16 18:21:04,645] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2024-11-16 18:21:04,645] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2024-11-16 18:21:04,645] [INFO] [logging.py:128:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2024-11-16 18:21:04,645] [INFO] [logging.py:128:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2024-11-16 18:21:04,664] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[2024-11-16 18:21:04,814] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning
[2024-11-16 18:21:04,815] [INFO] [utils.py:782:see_memory_usage] MA 31.57 GB         Max_MA 31.57 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:04,815] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 36.02 GB, percent = 2.4%
[2024-11-16 18:21:04,820] [INFO] [stage3.py:166:__init__] Reduce bucket size 500000000
[2024-11-16 18:21:04,820] [INFO] [stage3.py:167:__init__] Prefetch bucket size 50000000
[2024-11-16 18:21:04,947] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2024-11-16 18:21:04,948] [INFO] [utils.py:782:see_memory_usage] MA 31.57 GB         Max_MA 31.57 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:04,948] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 36.02 GB, percent = 2.4%
[2024-11-16 18:21:04,961] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
Tencent-L20:1157794:1158403 [0] NCCL INFO Using non-device net plugin version 0
Tencent-L20:1157794:1158403 [0] NCCL INFO Using network Socket
Tencent-L20:1157795:1158404 [1] NCCL INFO Using non-device net plugin version 0
Tencent-L20:1157795:1158404 [1] NCCL INFO Using network Socket
Tencent-L20:1157795:1158404 [1] NCCL INFO comm 0x127bcc20 rank 1 nranks 2 cudaDev 1 nvmlDev 4 busId 63000 commId 0xf4d40e2c2c0d3a71 - Init START
Tencent-L20:1157794:1158403 [0] NCCL INFO comm 0x11a24970 rank 0 nranks 2 cudaDev 0 nvmlDev 3 busId 43000 commId 0xf4d40e2c2c0d3a71 - Init START
Tencent-L20:1157794:1158403 [0] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
Tencent-L20:1157795:1158404 [1] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,00000000,00000000,00000000,00000000,00000000,00000000
Tencent-L20:1157795:1158404 [1] NCCL INFO comm 0x127bcc20 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
Tencent-L20:1157795:1158404 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
Tencent-L20:1157795:1158404 [1] NCCL INFO P2P Chunksize set to 131072
Tencent-L20:1157794:1158403 [0] NCCL INFO comm 0x11a24970 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
Tencent-L20:1157794:1158403 [0] NCCL INFO Channel 00/02 :    0   1
Tencent-L20:1157794:1158403 [0] NCCL INFO Channel 01/02 :    0   1
Tencent-L20:1157794:1158403 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
Tencent-L20:1157794:1158403 [0] NCCL INFO P2P Chunksize set to 131072
Tencent-L20:1157795:1158404 [1] NCCL INFO Channel 00 : 1[4] -> 0[3] via SHM/direct/direct
Tencent-L20:1157795:1158404 [1] NCCL INFO Channel 01 : 1[4] -> 0[3] via SHM/direct/direct
Tencent-L20:1157794:1158403 [0] NCCL INFO Channel 00 : 0[3] -> 1[4] via SHM/direct/direct
Tencent-L20:1157794:1158403 [0] NCCL INFO Channel 01 : 0[3] -> 1[4] via SHM/direct/direct
Tencent-L20:1157794:1158403 [0] NCCL INFO Connected all rings
Tencent-L20:1157795:1158404 [1] NCCL INFO Connected all rings
Tencent-L20:1157794:1158403 [0] NCCL INFO Connected all trees
Tencent-L20:1157795:1158404 [1] NCCL INFO Connected all trees
Tencent-L20:1157795:1158404 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
Tencent-L20:1157795:1158404 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
Tencent-L20:1157794:1158403 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
Tencent-L20:1157794:1158403 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
Tencent-L20:1157795:1158404 [1] NCCL INFO comm 0x127bcc20 rank 1 nranks 2 cudaDev 1 nvmlDev 4 busId 63000 commId 0xf4d40e2c2c0d3a71 - Init COMPLETE
Tencent-L20:1157794:1158403 [0] NCCL INFO comm 0x11a24970 rank 0 nranks 2 cudaDev 0 nvmlDev 3 busId 43000 commId 0xf4d40e2c2c0d3a71 - Init COMPLETE
Parameter Offload: Total persistent parameters: 42194240 in 1075 params
[2024-11-16 18:21:12,211] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2024-11-16 18:21:12,212] [INFO] [utils.py:782:see_memory_usage] MA 8.87 GB         Max_MA 31.57 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:12,212] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 58.43 GB, percent = 3.9%
[2024-11-16 18:21:12,370] [INFO] [utils.py:781:see_memory_usage] Before creating fp16 partitions
[2024-11-16 18:21:12,371] [INFO] [utils.py:782:see_memory_usage] MA 8.87 GB         Max_MA 8.87 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:12,371] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 58.43 GB, percent = 3.9%
[2024-11-16 18:21:12,753] [INFO] [utils.py:781:see_memory_usage] After creating fp16 partitions: 2
[2024-11-16 18:21:12,754] [INFO] [utils.py:782:see_memory_usage] MA 8.87 GB         Max_MA 8.87 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:12,754] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 59.26 GB, percent = 4.0%
[2024-11-16 18:21:12,913] [INFO] [utils.py:781:see_memory_usage] Before creating fp32 partitions
[2024-11-16 18:21:12,914] [INFO] [utils.py:782:see_memory_usage] MA 8.87 GB         Max_MA 8.87 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:12,914] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 59.26 GB, percent = 4.0%
[2024-11-16 18:21:13,261] [INFO] [utils.py:781:see_memory_usage] After creating fp32 partitions
[2024-11-16 18:21:13,261] [INFO] [utils.py:782:see_memory_usage] MA 8.87 GB         Max_MA 8.87 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:13,261] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 59.56 GB, percent = 4.0%
[2024-11-16 18:21:13,423] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2024-11-16 18:21:13,423] [INFO] [utils.py:782:see_memory_usage] MA 8.87 GB         Max_MA 8.87 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:13,423] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 60.05 GB, percent = 4.0%
[2024-11-16 18:21:13,815] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2024-11-16 18:21:13,815] [INFO] [utils.py:782:see_memory_usage] MA 8.87 GB         Max_MA 8.87 GB         CA 31.61 GB         Max_CA 32 GB
[2024-11-16 18:21:13,815] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 60.16 GB, percent = 4.0%
[2024-11-16 18:21:13,816] [INFO] [stage3.py:521:_setup_for_real_optimizer] optimizer state initialized
[2024-11-16 18:21:14,189] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2024-11-16 18:21:14,189] [INFO] [utils.py:782:see_memory_usage] MA 9.8 GB         Max_MA 9.81 GB         CA 32.54 GB         Max_CA 33 GB
[2024-11-16 18:21:14,190] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 60.55 GB, percent = 4.1%
[2024-11-16 18:21:14,190] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer_Stage3
[2024-11-16 18:21:14,190] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = None
[2024-11-16 18:21:14,190] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2024-11-16 18:21:14,190] [INFO] [logging.py:128:log_dist] [Rank 0] step=0, skipped=0, lr=[0.001, 0.001], mom=[(0.9, 0.999), (0.9, 0.999)]
[2024-11-16 18:21:14,194] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false,
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "synchronize_checkpoint_boundary": false,
    "profile": false
}
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   amp_params ................... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false,
    "start_step": null,
    "end_step": null,
    "metric_path": null,
    "arg_mappings": null,
    "metric": "throughput",
    "model_info": null,
    "results_dir": "autotuning_results",
    "exps_dir": "autotuning_exps",
    "overwrite": true,
    "fast": true,
    "start_profile_step": 3,
    "end_profile_step": 5,
    "tuner_type": "gridsearch",
    "tuner_early_stopping": 5,
    "tuner_num_trials": 50,
    "model_info_path": null,
    "mp_size": 1,
    "max_train_batch_size": null,
    "min_train_batch_size": 1,
    "max_train_micro_batch_size_per_gpu": 1.024000e+03,
    "min_train_micro_batch_size_per_gpu": 1,
    "num_tuning_micro_batch_sizes": 3
}
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f11ff123f50>
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   communication_data_type ...... torch.bfloat16
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   dump_state ................... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2024-11-16 18:21:14,194] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false,
    "recompute_fwd_factor": 0.0,
    "profile_step": 1,
    "module_depth": -1,
    "top_modules": 1,
    "detailed": true,
    "output_file": null
}
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   global_rank .................. 0
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 1
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   memory_breakdown ............. True
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false,
    "persistent_storage_path": null,
    "persistent_time_interval": 100,
    "num_of_version_in_retention": 2,
    "enable_nebula_load": true,
    "load_path": null
}
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   optimizer_name ............... None
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   optimizer_params ............. None
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   pld_params ................... False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   steps_per_print .............. 100
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   train_batch_size ............. 2
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  1
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... False
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   world_size ................... 2
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  True
[2024-11-16 18:21:14,195] [INFO] [config.py:1003:print]   zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme_path=None, buffer_count=5, buffer_size=100000000, max_in_cpu=1000000000, pin_memory=False) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True