Closed Youngluc closed 2 months ago
仔细看了一下,不是跨rank了,是在rank0内,从GPU1的pipe(layer0~layer20)到GPU2的pipe(layer21-34),tensor在传输过程中,序列长度发生变化了(2369->2262,这是为啥?
hi,想先问两个问题: 1.序列处理是否是按照本库的处理,因为我的preprocess不会出现不等长序列 2.错误的log和你的输出截图方便提供一下吗
---- 回复的原邮件 ---- 发件人 Hao cheng @.> 发送日期 2024年06月21日 16:47 收件人 Coobiw/MiniGPT4Qwen @.> 抄送人 Subscribed @.***> 主题 Re: [Coobiw/MiniGPT4Qwen] 哈喽打扰一下询问个问题! (Issue #25)
仔细看了一下,不是跨rank了,是在rank0内,从GPU1的pipe(layer0~layer20)到GPU2的pipe(layer21-34),tensor在传输过程中,序列长度发生变化了(2369->2262,这是为啥? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>
hi,想先问两个问题: 1.序列处理是否是按照本库的处理,因为我的preprocess不会出现不等长序列 2.错误的log和你的输出截图方便提供一下吗 … ---- 回复的原邮件 ---- 发件人 Hao cheng @.> 发送日期 2024年06月21日 16:47 收件人 Coobiw/MiniGPT4Qwen @.> 抄送人 Subscribed @.> 主题 Re: [Coobiw/MiniGPT4Qwen] 哈喽打扰一下询问个问题! (Issue #25) 仔细看了一下,不是跨rank了,是在rank0内,从GPU1的pipe(layer0~layer20)到GPU2的pipe(layer21-34),tensor在传输过程中,序列长度发生变化了(2369->2262,这是为啥? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.>
hi感谢您的回复,
RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([4, 2361, 6144]) and output[0] has a shape of torch.Size([4, 2481, 6144])
比较完整的错误报告(我打印了each layer的input_embeds.shape,还有我在collator中传入的一个tag【内容为input_ids.sum()与random.randint(100,2000)的一个拼接tensor】) `dynamic ViT batch size: 29, images per sample: 7.25, dynamic token length: 2262 [2024-06-21 16:40:50,826] [INFO] [checkpointing.py:539:forward] Activation Checkpointing Information [2024-06-21 16:40:50,826] [INFO] [checkpointing.py:540:forward] ----Partition Activations False, CPU CHECKPOINTING False [2024-06-21 16:40:50,826] [INFO] [checkpointing.py:541:forward] ----contiguous Memory Checkpointing False with None total layers [2024-06-21 16:40:50,826] [INFO] [checkpointing.py:543:forward] ----Synchronization False [2024-06-21 16:40:50,826] [INFO] [checkpointing.py:544:forward] ----Profiling time in checkpointing False torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.0 cuda:0 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.1 cuda:0 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.2 cuda:0 tensor([712564646, 1668], device='cuda:0') dynamic ViT batch size: 31, images per sample: 7.75, dynamic token length: 2361 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.3 cuda:0 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.0 cuda:1 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.4 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.1 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.5 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.2 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.6 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.3 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.7 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.4 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.8 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.5 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.9 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.6 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.10 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.7 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.11 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.8 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.12 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.9 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.13 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.10 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.14 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.11 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.15 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.12 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.16 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.13 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.17 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.14 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.18 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.15 cuda:1 tensor([712564646, 1668], device='cuda:0') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.19 cuda:0 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.16 cuda:1 tensor([712564646, 1668], device='cuda:0') [W ProcessGroupNCCL.cpp:1856] Warning: 0NCCL_AVOID_RECORD_STREAMS=1 has no effect for point-to-point collectives. (function operator()) tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.17 cuda:1 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.18 cuda:1 tensor([757955384, 1668], device='cuda:1') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.19 cuda:1 tensor([757955384, 1668], device='cuda:1') [W ProcessGroupNCCL.cpp:1856] Warning: 0NCCL_AVOID_RECORD_STREAMS=1 has no effect for point-to-point collectives. (function operator()) torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.20 cuda:2 tensor([712564646, 1668], device='cuda:2') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.21 cuda:2 tensor([712564646, 1668], device='cuda:2') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.22 cuda:2 tensor([712564646, 1668], device='cuda:2') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.23 cuda:2 tensor([712564646, 1668], device='cuda:2') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.24 cuda:2 torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.20 cuda:3 tensor([757955384, 1668], device='cuda:3') tensor([712564646, 1668], device='cuda:2') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.21 cuda:3 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.25 cuda:2 tensor([757955384, 1668], device='cuda:3') tensor([712564646, 1668], device='cuda:2') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.22 cuda:3 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.26 cuda:2 tensor([757955384, 1668], device='cuda:3') tensor([712564646, 1668], device='cuda:2') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.23 cuda:3 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.27 cuda:2 tensor([712564646, 1668], device='cuda:2') tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.28 cuda:2 torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.24 cuda:3 tensor([712564646, 1668], device='cuda:2') tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.29 cuda:2 torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.25 cuda:3 tensor([712564646, 1668], device='cuda:2') tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.30 cuda:2 torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.26 cuda:3 tensor([712564646, 1668], device='cuda:2') tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.31 cuda:2 torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.27 cuda:3 tensor([712564646, 1668], device='cuda:2') tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.32 cuda:2 torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.28 cuda:3 tensor([712564646, 1668], device='cuda:2') tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.33 cuda:2 torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.29 cuda:3 tensor([712564646, 1668], device='cuda:2') tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.30 cuda:3 tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.31 cuda:3 tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.32 cuda:3 tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.33 cuda:3 tensor([757955384, 1668], device='cuda:3') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.34 cuda:4 tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.35 cuda:4 tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.36 cuda:4 tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.37 cuda:4 tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.38 cuda:4 tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.39 cuda:4 torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.34 cuda:5 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.35 cuda:5 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.40 cuda:4 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.36 cuda:5 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.41 cuda:4 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.37 cuda:5 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.42 cuda:4 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.38 cuda:5 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.43 cuda:4 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.39 cuda:5 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.44 cuda:4 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.40 cuda:5 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.45 cuda:4 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.41 cuda:5 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.46 cuda:4 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.42 cuda:5 torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.47 cuda:4 tensor([757955384, 1668], device='cuda:5') tensor([712564646, 1668], device='cuda:4') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.43 cuda:5 tensor([757955384, 1668], device='cuda:5') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.44 cuda:5 tensor([757955384, 1668], device='cuda:5') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.45 cuda:5 tensor([757955384, 1668], device='cuda:5') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.46 cuda:5 tensor([757955384, 1668], device='cuda:5') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.47 cuda:5 tensor([757955384, 1668], device='cuda:5') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.48 cuda:6 tensor([712564646, 1668], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.49 cuda:6 tensor([712564646, 1668], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.50 cuda:6 tensor([712564646, 1668], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.51 cuda:6 tensor([712564646, 1668], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.52 cuda:6 tensor([712564646, 1668], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.53 cuda:6 tensor([712564646, 1668], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.54 cuda:6 tensor([712564646, 1668], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.55 cuda:6 tensor([712564646, 1668], device='cuda:6') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.48 cuda:7 tensor([757955384, 1668], device='cuda:7') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.49 cuda:7 tensor([757955384, 1668], device='cuda:7') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.50 cuda:7 tensor([757955384, 1668], device='cuda:7') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.51 cuda:7 tensor([757955384, 1668], device='cuda:7') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.52 cuda:7 tensor([757955384, 1668], device='cuda:7') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.53 cuda:7 tensor([757955384, 1668], device='cuda:7') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.54 cuda:7 tensor([757955384, 1668], device='cuda:7') torch.Size([4, 2361, 6144]) InternLMBlockPipeLayer.55 cuda:7 tensor([757955384, 1668], device='cuda:7') [W ProcessGroupNCCL.cpp:1856] Warning: 0NCCL_AVOID_RECORD_STREAMS=1 has no effect for point-to-point collectives. (function operator()) [W ProcessGroupNCCL.cpp:1856] Warning: 0NCCL_AVOID_RECORD_STREAMS=1 has no effect for point-to-point collectives. (function operator()) [2024-06-21 16:41:08,780] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_allgather: 10.02 | optimizer_gradients: 32.98 | optimizer_step: 89.77 0 batch end 0 batch end 0 batch end 0 batch end 0 batch end 0 batch end 0 batch end 0 batch end 06/21/2024 16:41:11 - INFO - main - {'loss': 2.441850185394287, 'learning_rate': 0.0, 'epoch': 0.0}
Epoch 1: 0%| | 0/15698 [00:26<?, ?it/s, loss=2.44, learning_rate=0, epoch=0][A
Epoch 1: 0%| | 1/15698 [00:26<115:52:37, 26.58s/it, loss=2.44, learning_rate=0, epoch=0][Adynamic ViT batch size: 31, images per sample: 7.75, dynamic token length: 2481
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.0 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.1 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.2 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.3 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.4 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.5 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.6 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.7 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.8 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.9 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.10 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.11 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.12 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.13 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.14 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.15 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.16 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.17 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.18 cuda:1 tensor([766560813, 1374], device='cuda:1')
torch.Size([4, 2481, 6144]) InternLMBlockPipeLayer.19 cuda:1 tensor([766560813, 1374], device='cuda:1')
Traceback (most recent call last):
File "/mnt/workspace/workgroup/chenghao/video_analysis/internvl_chat_interleaved/internvl/train/intern_vl_chat_finetune_block_pp.py", line 850, in
Epoch 1: 0%| | 1/15698 [00:33<146:05:44, 33.51s/it, loss=2.44, learning_rate=0, epoch=0]
Traceback (most recent call last):
File "/mnt/workspace/workgroup/chenghao/video_analysis/internvl_chat_interleaved/internvl/train/intern_vl_chat_finetune_block_pp.py", line 850, in
Epoch: 0it [02:15, ?it/s] torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.33 cuda:2 tensor([790315515, 1374], device='cuda:2') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.34 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.35 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.36 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.37 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.38 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.39 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.40 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.41 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.42 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.43 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.44 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.45 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.46 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.47 cuda:4 tensor([790315515, 1374], device='cuda:4') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.48 cuda:6 tensor([790315515, 1374], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.49 cuda:6 tensor([790315515, 1374], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.50 cuda:6 tensor([790315515, 1374], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.51 cuda:6 tensor([790315515, 1374], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.52 cuda:6 tensor([790315515, 1374], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.53 cuda:6 tensor([790315515, 1374], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.54 cuda:6 tensor([790315515, 1374], device='cuda:6') torch.Size([4, 2262, 6144]) InternLMBlockPipeLayer.55 cuda:6 tensor([790315515, 1374], device='cuda:6') [2024-06-21 16:41:30,578] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 225308 [2024-06-21 16:41:30,578] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 225309 [2024-06-21 16:41:31,133] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 225310`
模型的一个pipeline切分: [2024-06-21 16:38:26,030] [INFO] [module.py:375:_partition_layers] Partitioning pipeline stages with method parameters stage=0 layers=21 0: TokenizerPipeLayer 1: InternLMBlockPipeLayer 2: InternLMBlockPipeLayer 3: InternLMBlockPipeLayer 4: InternLMBlockPipeLayer 5: InternLMBlockPipeLayer 6: InternLMBlockPipeLayer 7: InternLMBlockPipeLayer 8: InternLMBlockPipeLayer 9: InternLMBlockPipeLayer 10: InternLMBlockPipeLayer 11: InternLMBlockPipeLayer 12: InternLMBlockPipeLayer 13: InternLMBlockPipeLayer 14: InternLMBlockPipeLayer 15: InternLMBlockPipeLayer 16: InternLMBlockPipeLayer 17: InternLMBlockPipeLayer 18: InternLMBlockPipeLayer 19: InternLMBlockPipeLayer 20: InternLMBlockPipeLayer stage=1 layers=14 21: InternLMBlockPipeLayer 22: InternLMBlockPipeLayer 23: InternLMBlockPipeLayer 24: InternLMBlockPipeLayer 25: InternLMBlockPipeLayer 26: InternLMBlockPipeLayer 27: InternLMBlockPipeLayer 28: InternLMBlockPipeLayer 29: InternLMBlockPipeLayer 30: InternLMBlockPipeLayer 31: InternLMBlockPipeLayer 32: InternLMBlockPipeLayer 33: InternLMBlockPipeLayer 34: InternLMBlockPipeLayer stage=2 layers=14 35: InternLMBlockPipeLayer 36: InternLMBlockPipeLayer 37: InternLMBlockPipeLayer 38: InternLMBlockPipeLayer 39: InternLMBlockPipeLayer 40: InternLMBlockPipeLayer 41: InternLMBlockPipeLayer 42: InternLMBlockPipeLayer 43: InternLMBlockPipeLayer 44: InternLMBlockPipeLayer 45: InternLMBlockPipeLayer 46: InternLMBlockPipeLayer 47: InternLMBlockPipeLayer 48: InternLMBlockPipeLayer stage=3 layers=11 49: InternLMBlockPipeLayer 50: InternLMBlockPipeLayer 51: InternLMBlockPipeLayer 52: InternLMBlockPipeLayer 53: InternLMBlockPipeLayer 54: InternLMBlockPipeLayer 55: InternLMBlockPipeLayer 56: InternLMBlockPipeLayer 57: FLNPipeLayer 58: LMPipeLayer 59: LossPipeLayer
会发现序列长度为2369和2481的这两个序列,好像过了stage0(layer19)之后就被阻断了,然后这个grad_tensor和output之间的match关系也比较混乱。。。
在log里会有一个告诉你每个stage分在哪个rank上的输出 方便也share一下吗
---- 回复的原邮件 ---- 发件人 Hao cheng @.> 发送日期 2024年06月21日 17:03 收件人 Coobiw/MiniGPT4Qwen @.> 抄送人 Coobiw @.>, Comment @.> 主题 Re: [Coobiw/MiniGPT4Qwen] 哈喽打扰一下询问个问题! (Issue #25)
模型的一个pipeline切分: [2024-06-21 16:38:26,030] [INFO] [module.py:375:_partition_layers] Partitioning pipeline stages with method parameters stage=0 layers=21 0: TokenizerPipeLayer 1: InternLMBlockPipeLayer 2: InternLMBlockPipeLayer 3: InternLMBlockPipeLayer 4: InternLMBlockPipeLayer 5: InternLMBlockPipeLayer 6: InternLMBlockPipeLayer 7: InternLMBlockPipeLayer 8: InternLMBlockPipeLayer 9: InternLMBlockPipeLayer 10: InternLMBlockPipeLayer 11: InternLMBlockPipeLayer 12: InternLMBlockPipeLayer 13: InternLMBlockPipeLayer 14: InternLMBlockPipeLayer 15: InternLMBlockPipeLayer 16: InternLMBlockPipeLayer 17: InternLMBlockPipeLayer 18: InternLMBlockPipeLayer 19: InternLMBlockPipeLayer 20: InternLMBlockPipeLayer stage=1 layers=14 21: InternLMBlockPipeLayer 22: InternLMBlockPipeLayer 23: InternLMBlockPipeLayer 24: InternLMBlockPipeLayer 25: InternLMBlockPipeLayer 26: InternLMBlockPipeLayer 27: InternLMBlockPipeLayer 28: InternLMBlockPipeLayer 29: InternLMBlockPipeLayer 30: InternLMBlockPipeLayer 31: InternLMBlockPipeLayer 32: InternLMBlockPipeLayer 33: InternLMBlockPipeLayer 34: InternLMBlockPipeLayer stage=2 layers=14 35: InternLMBlockPipeLayer 36: InternLMBlockPipeLayer 37: InternLMBlockPipeLayer 38: InternLMBlockPipeLayer 39: InternLMBlockPipeLayer 40: InternLMBlockPipeLayer 41: InternLMBlockPipeLayer 42: InternLMBlockPipeLayer 43: InternLMBlockPipeLayer 44: InternLMBlockPipeLayer 45: InternLMBlockPipeLayer 46: InternLMBlockPipeLayer 47: InternLMBlockPipeLayer 48: InternLMBlockPipeLayer stage=3 layers=11 49: InternLMBlockPipeLayer 50: InternLMBlockPipeLayer 51: InternLMBlockPipeLayer 52: InternLMBlockPipeLayer 53: InternLMBlockPipeLayer 54: InternLMBlockPipeLayer 55: InternLMBlockPipeLayer 56: InternLMBlockPipeLayer 57: FLNPipeLayer 58: LMPipeLayer 59: LossPipeLayer 会发现序列长度为2369和2481的这两个序列,好像过了stage1之后就被阻断了,然后这个grad_tensor和output之间的match关系也比较混乱。。。 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
感觉你的划分应该是1357 0246,然后out普遍是长的那个(其实就是batch里的max_length),grad应该是正确的长度,我感觉你可以检查下你的collator、每个block的input和output,注意一下符合deepspeed的pipeline module的协议,尽可能都以tensor的形式传输。然后注意pipelinemodel会有一个label项作为输入(以tuple形式),检查一下 比如collator大概的返回形式应该和这个函数类似:
def collate_fn_minigpt4qwen(batch,preprocess_func):
image_list, conversation_list = [], []
for sample in batch:
image_list.append(sample["image"])
conversation_list.append(sample["conversations"])
new_batch = \
{
"image": torch.stack(image_list, dim=0),
"conversations": conversation_list,
}
data_dict = preprocess_func(new_batch['conversations'])
return ((new_batch['image'], data_dict['input_ids'],data_dict['labels'],data_dict['attention_mask']),
data_dict['labels']
) # 我这里是Tuple[Tuple[Tensor], Tensor]
可以参考下我的这个博客里踩过的一些坑:https://zhuanlan.zhihu.com/p/684462477
感觉你的划分应该是1357 0246,然后out普遍是长的那个(其实就是batch里的max_length),grad应该是正确的长度,我感觉你可以检查下你的collator、每个block的input和output,注意一下符合deepspeed的pipeline module的协议,尽可能都以tensor的形式传输。然后注意pipelinemodel会有一个label项作为输入(以tuple形式),检查一下 比如collator大概的返回形式应该和这个函数类似:
def collate_fn_minigpt4qwen(batch,preprocess_func): image_list, conversation_list = [], [] for sample in batch: image_list.append(sample["image"]) conversation_list.append(sample["conversations"]) new_batch = \ { "image": torch.stack(image_list, dim=0), "conversations": conversation_list, } data_dict = preprocess_func(new_batch['conversations']) return ((new_batch['image'], data_dict['input_ids'],data_dict['labels'],data_dict['attention_mask']), data_dict['labels'] ) # 我这里是Tuple[Tuple[Tensor], Tensor]
可以参考下我的这个博客里踩过的一些坑:https://zhuanlan.zhihu.com/p/684462477
对的,stage划分是1357 0246这样,collator最后返回的是按照大佬博客里讲的Tuple[Tuple[torch.Tensor], Any]这样的形式,我仔细研究了一下,测试的时候发现是,出错的那几个batch,activation在从GPU0(GPU1)转移到GPU2(GPU3)的时候,出现问题,变成这个rank上个step的batch的序列形状了。我举个例子: 比如在rank1(GPU0246)上,第一个batch的input_embeds是(4,2262,6144)的形状,第二个batch是(4,2361,6144)的形状。第一个batch的forward和backward是正常的,而第二个batch在GPU0上是正常的,GPU0上每个layer的hidden_states形状都是(4,2361,6144),但是到GPU2上的layer时,形状就都变成(4,2262,6144)了,然后就会报错: RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([4, 2262, 6144]) and output[0] has a shape of torch.Size([4, 2361, 6144]). 这是实际情况,我很纳闷为什么会这样。我改变了pp的等级(pp=2/4/8),都是上述这种情况,第一个step都是正常的,第二个开始就会这样了。大佬可以帮忙分析一下为什么吗?非常感谢!!!(如果太占用大佬时间就算惹
感觉你的划分应该是1357 0246,然后out普遍是长的那个(其实就是batch里的max_length),grad应该是正确的长度,我感觉你可以检查下你的collator、每个block的input和output,注意一下符合deepspeed的pipeline module的协议,尽可能都以tensor的形式传输。然后注意pipelinemodel会有一个label项作为输入(以tuple形式),检查一下 比如collator大概的返回形式应该和这个函数类似:
def collate_fn_minigpt4qwen(batch,preprocess_func): image_list, conversation_list = [], [] for sample in batch: image_list.append(sample["image"]) conversation_list.append(sample["conversations"]) new_batch = \ { "image": torch.stack(image_list, dim=0), "conversations": conversation_list, } data_dict = preprocess_func(new_batch['conversations']) return ((new_batch['image'], data_dict['input_ids'],data_dict['labels'],data_dict['attention_mask']), data_dict['labels'] ) # 我这里是Tuple[Tuple[Tensor], Tensor]
可以参考下我的这个博客里踩过的一些坑:https://zhuanlan.zhihu.com/p/684462477
训练的第一个step应该是正常的,第二个step就有问题了,第二个step的batch只能在stage0对应的GPU上传播正常,从stage0转移到stage1的GPU上时,hidden_states的形状就变为step1中的形状了。大佬的ds版本和torch版本可以说一下吗?我现在搞不清楚这个问题的真正根源是在哪。
方便看下你block的代码吗,或者你知乎私我个联系方式 有空一起看一下子?
---- 回复的原邮件 ---- 发件人 Hao cheng @.> 发送日期 2024年06月21日 23:39 收件人 Coobiw/MiniGPT4Qwen @.> 抄送人 Coobiw @.>, Comment @.> 主题 Re: [Coobiw/MiniGPT4Qwen] 哈喽打扰一下询问个问题! (Issue #25)
感觉你的划分应该是1357 0246,然后out普遍是长的那个(其实就是batch里的max_length),grad应该是正确的长度,我感觉你可以检查下你的collator、每个block的input和output,注意一下符合deepspeed的pipeline module的协议,尽可能都以tensor的形式传输。然后注意pipelinemodel会有一个label项作为输入(以tuple形式),检查一下 比如collator大概的返回形式应该和这个函数类似: def collate_fn_minigpt4qwen(batch,preprocess_func): image_list, conversation_list = [], []
for sample in batch:
image_list.append(sample["image"])
conversation_list.append(sample["conversations"])
new_batch = \
{
"image": torch.stack(image_list, dim=0),
"conversations": conversation_list,
}
data_dict = preprocess_func(new_batch['conversations'])
return ((new_batch['image'], data_dict['input_ids'],data_dict['labels'],data_dict['attention_mask']),
data_dict['labels']
) # 我这里是Tuple[Tuple[Tensor], Tensor]
可以参考下我的这个博客里踩过的一些坑:https://zhuanlan.zhihu.com/p/684462477 训练的第一个step应该是正常的,第二个step就有问题了,第二个step的batch只能在stage0对应的GPU上传播正常,从stage0转移到stage1的GPU上时,hidden_states的形状就变为step1中的形状了。大佬的ds版本和torch版本可以说一下吗?我现在搞不清楚这个问题的真正根源是在哪。 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
方便看下你block的代码吗,或者你知乎私我个联系方式 有空一起看一下子? … ---- 回复的原邮件 ---- 发件人 Hao cheng @.> 发送日期 2024年06月21日 23:39 收件人 Coobiw/MiniGPT4Qwen @.> 抄送人 Coobiw @.>, Comment @.> 主题 Re: [Coobiw/MiniGPT4Qwen] 哈喽打扰一下询问个问题! (Issue #25) 感觉你的划分应该是1357 0246,然后out普遍是长的那个(其实就是batch里的max_length),grad应该是正确的长度,我感觉你可以检查下你的collator、每个block的input和output,注意一下符合deepspeed的pipeline module的协议,尽可能都以tensor的形式传输。然后注意pipelinemodel会有一个label项作为输入(以tuple形式),检查一下 比如collator大概的返回形式应该和这个函数类似: def collate_fn_minigpt4qwen(batch,preprocess_func): image_list, conversation_list = [], [] for sample in batch: image_list.append(sample["image"]) conversation_list.append(sample["conversations"]) new_batch = \ { "image": torch.stack(image_list, dim=0), "conversations": conversation_list, } data_dict = preprocess_func(new_batch['conversations']) return ((new_batch['image'], data_dict['input_ids'],data_dict['labels'],data_dict['attention_mask']), data_dict['labels'] ) # 我这里是Tuple[Tuple[Tensor], Tensor] 可以参考下我的这个博客里踩过的一些坑:https://zhuanlan.zhihu.com/p/684462477 训练的第一个step应该是正常的,第二个step就有问题了,第二个step的batch只能在stage0对应的GPU上传播正常,从stage0转移到stage1的GPU上时,hidden_states的形状就变为step1中的形状了。大佬的ds版本和torch版本可以说一下吗?我现在搞不清楚这个问题的真正根源是在哪。 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
好好好,我知乎私聊大佬!代码的话是在办公笔记本上我没法直接复制粘贴😭
Solved.
We find that DeepSpeed Pipeline Parallel needs the same seq_length
in a mini-batch(including many micro-batch) and the same batch-size(so we should set drop_last
to True
).
This is a good discovery. I'll close but pin this issue.
网上关于Deepspeed流水线并行的资料实在太少了...我遇到个问题需要请教一下,麻烦大佬有时间帮忙分析一下... 我仿照大佬的代码写了一个其他VLM的training code,在训练过程中会遇到一个奇怪的问题: setting如下:num_stages=4, ngpus_per_node=8,那么pp=4,dp=2,然后rank0和rank1会分别有两个batch:B1和B2,假设B1和B2的序列长度分别为N1和N2。 然后在autograd时候就出错了,说的是Mismatch shape错误,grad的shape为N1,output的shape为N2,相当于autograd时候用了rank1的batch B1去更新rank2的batch B2了。
这个BUG或者问题我实在无从下手解决,也没搜集到相关资料。
P.S.: 我在LLM的BlockPipeLayer中打印了一下,发现B1的数据完整的forward了所有层,B2的数据只forward了前20多个层,后面的层还没传播完。是不是哪里的同步有问题啊?