search

DLLXW / baby-llama2-chinese

用于从头预训练+SFT一个小参数量的中文LLaMa2的仓库；24G单卡即可运行得到一个具备简单中文问答能力的chat-llama2.
MIT License
2.47k stars 305 forks source link

Ignore the `freqs_cis` buffer so that DDP does not broadcast it at construction time #59

Open xiaoguzai opened 7 months ago

xiaoguzai commented 7 months ago

在pretrain中有这样一部分代码

if ddp:
    # Ignore the `freqs_cis` buffer so that DDP does not broadcast it at
    # construction time since NCCL does not support `ComplexFloat`
    prefix = "_orig_mod." if compile else ""
    model._ddp_params_and_buffers_to_ignore = {prefix + "freqs_cis"}
    model = DDP(model, device_ids=[ddp_local_rank])

这里的操作作用是什么？有没有大佬指点一下？