the performance isn't as expected with nvidia-tensorflow in A100

DeepRec-AI / DeepRec

DeepRec is a high-performance recommendation deep learning framework based on TensorFlow. It is hosted in incubation in LF AI & Data Foundation.

Apache License 2.0

1.03k stars 349 forks source link

the performance isn't as expected with nvidia-tensorflow in A100 #110

Open thinking03 opened 2 years ago

thinking03 commented 2 years ago

测试环境：A100 单机4卡horovod同步训练 DeepRec vs Nvidia-TF 1.15.4 训练样本：3路embedding，样本存储在hdfs，tfrecord结构 batch_size 1024 模型结构：transformer+mmoe 测试性能：DeepRec 1.8s/step Nvidia-TF 0.55s/step

liutongxuan commented 2 years ago

Please use the latest DeepRec to compare the performance.

liutongxuan commented 2 years ago

We have test all the models in modelzoo, compare with NV-TF in A100: