Add mlperf benchmark scripts in-tree. - Githubissues

google / jetstream-pytorch

PyTorch/XLA integration with JetStream (https://github.com/google/JetStream) for LLM inference"

Apache License 2.0

33 stars 14 forks source link

Add mlperf benchmark scripts in-tree. #148

Closed qihqi closed 2 months ago

qihqi commented 2 months ago

TODOS:

try llama70.
Modify backend.py to use asyncio instead of threadpool.

=== All python files are copied from https://github.com/tpu-inference/inference_mlperf4.1/tree/mixtral_loadgen/language/mixtral-8x7b and shell scripts adapted from https://docs.google.com/document/d/112oYiFB_hkbb0_Kfcm0iDaDVcsq3JwLYprc_1zh_FCo/edit?tab=t.0

With the following changes:

dont install mlperf loadgen from source but install from pip instead
install scripts / run scripts respects virtualenv
remove 'warmup' feature from loadgen scripts; but do it in a separate script intead.