火山引擎 libai_gpt 与 megatron_gpt 对比测试

[Tendo33]

• GPT-2

  	libai	Megatron
  数据集	loss_compara_content_sentence.bin loss_compara_content_sentence.idx	loss_compara_content_sentence.bin loss_compara_content_sentence.idx
  vocab.txt	bert-base-chinese-vocab.txt	bert-base-chinese-vocab.txt
  测试脚本	args_train.sh	megatron_args_pretrain_gpt2.sh

• 测试环境

  OneFlow	Libai	Megatron
  (master分支)0d4bc37	(main分支)f728a5ec	(mian分支)e156d2f	libai开启nccl_use_compute_stream

• 测试结果

NVIDIA A100-SXM4-80GB	OneFlow_0d4bc37	Megatron_e156d2f
gpt2_pretrain_graph_nl60_nah144_hs2304_FP16actrue DP16_MP1_PP1_zerofalse_stage2_mbs2_gbs32_acc1_2n8g	80290-80370 MiB /29.15 samples/s	77944 - 78072 MiB /26.4 samples/s
gpt2_pretrain_graph_nl60_nah144_hs2304_FP16actrue DP16_MP1_PP1_zerofalse_stage2_mbs2_gbs32_acc4_2n8g	78316-78573 MiB/33 samples/s	77944 - 78072 MiB /33.8 samples/s