Open macheng6 opened 2 months ago
use_varlen_attn=True
感谢回答,还有疑问是:
use_varlen_attn=True时,报错如下: File "/opt/conda/lib/python3.8/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 748, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl result = forward_call(*args, **kwargs) File "/nas/macheng.ma/projects/xtuner-main/xtuner/model/modules/dispatch/qwen2.py", line 317, in qwen2_varlen_attn_forward attn_output = varlen_flash_attn(attn_output = varlen_flash_attn(
File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/attention.py", line 65, in sequence_parallel_attn attn_output = varlen_flash_attn( File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/attention.py", line 65, in sequence_parallel_attn
File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/attention.py", line 65, in sequence_parallel_attn pre_process_for_sequence_parallel_attn(pre_process_for_sequence_parallel_attn(
File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/attention.py", line 23, in pre_process_for_sequence_parallel_attn File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/attention.py", line 23, in pre_process_for_sequence_parallel_attn pre_process_for_sequence_parallel_attn( File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/attention.py", line 23, in pre_process_for_sequence_parallel_attn query_states = all_to_all( File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 87, in all_to_all query_states = all_to_all( File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 87, in all_to_all query_states = all_to_all( File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 87, in all_to_all return _AllToAll.apply(input, sp_group, scatter_dim, gather_dim) File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply return _AllToAll.apply(input, sp_group, scatter_dim, gather_dim) File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply return _AllToAll.apply(input, sp_group, scatter_dim, gather_dim) File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply return super().apply(*args, kwargs) # type: ignore[misc] File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 42, in forward return super().apply(*args, *kwargs) # type: ignore[misc] File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 42, in forward return super().apply(args, kwargs) # type: ignore[misc] File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 42, in forward output = _all_to_all(input, ctx.world_size, sp_group, scatter_dim, File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 21, in _all_to_all output = _all_to_all(input, ctx.world_size, sp_group, scatter_dim, File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 21, in _all_to_all output = _all_to_all(input, ctx.world_size, sp_group, scatter_dim, File "/nas/macheng.ma/projects/xtuner-main/xtuner/parallel/sequence/comm.py", line 21, in _all_to_all dist.all_to_all(output_list, input_list, group=group) File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1436, in wrapper dist.all_to_all(output_list, input_list, group=group) File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1436, in wrapper dist.all_to_all(output_list, input_list, group=group) File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1441, in wrapper return func(*args, *kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3266, in all_to_all "args": f"{args}, {kwargs}",return func(args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 426, in repr File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3266, in all_to_all return torch._tensor_str._str(self, tensor_contents=tensor_contents) File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 636, in _str work = group.alltoall(output_tensor_list, input_tensor_list, opts) torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1275, internal error, NCCL version 2.14.3 ncclInternalError: Internal check failed.
qwen1_5_0_5b_chat_full_alpaca_e3.py这个文件中,by_epoch=True应该是按照epoch来更新学习率,并不是按照step?
好的,感谢回答。
使用A100-32卡,xtuner训练qwen1.5-32b长序列,有下面几个疑问想请教一下: