Open ammerser opened 1 year ago
INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [1,2,3,4] LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [1,2,3,4] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1,2,3,4] LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [1,2,3,4]
27.8 M Trainable params 1.1 M Non-trainable params 28.9 M Total params 115.404 Total estimated model params size (MB) Epoch 0: 8%|███████▍ | 5/65 [00:03<00:36, 1.62it/s, loss=3.99, v_num=8, loss_step=3.990cuda:0 4 {'acc': 0.02}███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 6.10it/s] 段错误 (核心已转储)
INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [1,2,3,4] LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [1,2,3,4] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1,2,3,4] LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [1,2,3,4]
| Name | Type | Params
0 | sed_model | HTSAT_Swin_Transformer | 28.9 M
27.8 M Trainable params 1.1 M Non-trainable params 28.9 M Total params 115.404 Total estimated model params size (MB) Epoch 0: 8%|███████▍ | 5/65 [00:03<00:36, 1.62it/s, loss=3.99, v_num=8, loss_step=3.990cuda:0 4 {'acc': 0.02}███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 6.10it/s] 段错误 (核心已转储)