issues
search
Tendo33
/
oneflow-test
oneflow test
0
stars
0
forks
source link
火山引擎 libai_gpt 与 megatron_gpt 对比测试
#13
Open
Tendo33
opened
1 year ago
Tendo33
commented
1 year ago
GPT-2
libai
Megatron
数据集
loss_compara_content_sentence.bin
loss_compara_content_sentence.idx
loss_compara_content_sentence.bin
loss_compara_content_sentence.idx
vocab.txt
bert-base-chinese-vocab.txt
bert-base-chinese-vocab.txt
测试脚本
args_train.sh
megatron_args_pretrain_gpt2.sh
测试环境
OneFlow
Libai
Megatron
(master分支)
0d4bc37
(main分支)
f728a5ec
(mian分支)
e156d2f
libai开启nccl_use_compute_stream
测试结果
NVIDIA A100-SXM4-80GB
OneFlow_0d4bc37
Megatron_e156d2f
gpt2_pretrain_graph_nl60_nah144_hs2304_FP16
actrue
DP16_MP1_PP1_zerofalse_stage2_mbs2_gbs32_acc1_2n8g
80290-80370 MiB
/
29.15 samples/s
77944 - 78072 MiB
/
26.4 samples/s
gpt2_pretrain_graph_nl60_nah144_hs2304_FP16
actrue
DP16_MP1_PP1_zerofalse_stage2_mbs2_gbs32_acc4_2n8g
78316-78573 MiB
/
33 samples/s
77944 - 78072 MiB
/
33.8 samples/s
GPT-2
loss_compara_content_sentence.idx
loss_compara_content_sentence.idx
测试环境
测试结果
DP16_MP1_PP1_zerofalse_stage2_mbs2_gbs32_acc1_2n8g
DP16_MP1_PP1_zerofalse_stage2_mbs2_gbs32_acc4_2n8g