search

amd / ZenDNN

Other
82 stars 14 forks source link

DLRM suggest benchmark #4

Open JunxiChhen opened 2 years ago

JunxiChhen commented 2 years ago

Hi, I read about your doc and the example of benchmark only has CNN related model. And what benchmark tool do you suggest to use with DLRM and running on AMD platform using ZenDNN?

Thanks

ratan-prasad commented 2 years ago

ZenDNN library is intended to be used in conjunction with the frameworks like TensorFlow, PyTorch, ONNXRT and cannot be used independently. ZenDNN release v3.3 is available at https://developer.amd.com/zendnn/

We recommend to refer the official DLRM https://github.com/facebookresearch/dlrm.git for benchmarking.

JunxiChhen commented 2 years ago

Thank you! It is really helpful. BTW, how about the suggest benchmark of Bert-Large?

JunxiChhen commented 2 years ago

ZenDNN library is intended to be used in conjunction with the frameworks like TensorFlow, PyTorch, ONNXRT and cannot be used independently. ZenDNN release v3.3 is available at https://developer.amd.com/zendnn/

We recommend to refer the official DLRM https://github.com/facebookresearch/dlrm.git for benchmarking.

When I ran the official dlrm benchmark with ZenDNN v3.3 pytorch (python3.8), there is an issue: RuntimeError: could not execute a primitive And when I use previous ZenDNN version (v3.3), there is a core dumped issue: Segmentation fault (core dumped) The issue happened when min-batch-size equal and larger than 128 on my test vm (m6a.16xlarge).

These issues didn't happen when I didn't use ZenDNN. So are there any tricks to avoid them?

The benchmark cmd: export OMP_NUM_THREADS=8 && export GOMP_CPU_AFFINITY=0-7 && numactl --physcpubind=0-7 -m 0 python dlrm_s_pytorch.py --mini-batch-size=128 --test-num-workers=0 --num-batches=100 --data-generation=random --arch-mlp-bot=512-512-64 --arch-mlp-top=1024-1024-1024-1 --arch-sparse-feature-size=64 --arch-embedding-size=1000000-1000000-1000000-1000000-1000000-1000000-1000000-1000000 --num-indices-per-lookup=100 --arch-interaction-op=dot --numpy-rand-seed=727 --print-freq=100 --print-time --inference-only

JunxiChhen commented 2 years ago

Segmentation fault (core dumped) occurred using ZenDNN3.2 pytorch when using criteo data benchmark on dlrm_s_pytorch.py (in https://github.com/facebookresearch/dlrm.git)

ratan-prasad commented 2 years ago

Thank you! It is really helpful. BTW, how about the suggest benchmark of Bert-Large?

Hi, Please refer the Bert: https://github.com/google-research/bert

ratan-prasad commented 2 years ago

Segmentation fault (core dumped) occurred using ZenDNN3.2 pytorch when using criteo data benchmark on dlrm_s_pytorch.py (in https://github.com/facebookresearch/dlrm.git) Hi, we found the issue when running official dlrm benchmark and criteo data benchmark. We will submit the fix very soon.

JunxiChhen commented 2 years ago

Segmentation fault (core dumped) occurred using ZenDNN3.2 pytorch when using criteo data benchmark on dlrm_s_pytorch.py (in https://github.com/facebookresearch/dlrm.git) Hi, we found the issue when running official dlrm benchmark and criteo data benchmark. We will submit the fix very soon.

Thank you. And how about the "RuntimeError: could not execute a primitive" using zendnn3.3?

ratan-prasad commented 2 years ago

Segmentation fault (core dumped) occurred using ZenDNN3.2 pytorch when using criteo data benchmark on dlrm_s_pytorch.py (in https://github.com/facebookresearch/dlrm.git) Hi, we found the issue when running official dlrm benchmark and criteo data benchmark. We will submit the fix very soon.

Thank you. And how about the "RuntimeError: could not execute a primitive" using zendnn3.3?

That is also fixed. The fix will be submitted very soon.