Open bin1guo opened 4 months ago
test opt 1.3 model on EMR platform with 52c. the performance is not right with BS=1. the gap between BS=1 and BS =2 is too big.
numactl -C 0-51 -m 0 ./run_benchmark.sh -m opt-1_3b -d bf16 -s 1 -bs 1 -in 128 -out 15 -i 10
the results BS=1
BS=2
@bin1guo do we still need to benchmark OPT model? Suggest to run the llama model.
test opt 1.3 model on EMR platform with 52c. the performance is not right with BS=1. the gap between BS=1 and BS =2 is too big.
numactl -C 0-51 -m 0 ./run_benchmark.sh -m opt-1_3b -d bf16 -s 1 -bs 1 -in 128 -out 15 -i 10
the results BS=1![image](https://github.com/intel/xFasterTransformer/assets/90182771/aeaa4347-b61b-4463-b057-c600011dbc8e)
BS=2![image](https://github.com/intel/xFasterTransformer/assets/90182771/c88bb66b-2e75-47d4-b40b-076bcee99eed)