Closed JackCaoG closed 3 months ago
raghul-tonita
commented
3 months ago s/--model_name=/--model
There is a bug in the readme, the argument is --model.
There is no argument called model_name, you should remove it from the command.
The sample command has --model affixed to the right, so you will need to just remove the --model_name param.
JackCaoG
commented
3 months ago Thanks @raghul-tonita if I want to benchmark the throughput what should I do? Do I need to first start the server and then run the benchmark script?
raghul-tonita
commented
3 months ago Yes. I had set up the server and load tested using a variant of https://github.com/google/JetStream/blob/main/jetstream/tools/load_tester.py. In short, setup the server and sent async grpc requests with my custom dataset to benchmark.
You could also use the same dataset that was used to generate the benchmarks report in the benchmarks.md and run the benchmarking script -- I haven't done this, so I might not know.
JackCaoG
commented
3 months ago Thanks @raghul-tonita I am able to start the server with
python run_server.py --param_size=7b --batch_size=64 --max_cache_length=2048 --quantize_weights=$quantize --quantize_kv_cache=$quantize --checkpoint_path=$output_ckpt_dir --tokenizer_path=$tokenizer_path --platform=tpu=4 --model=llama-2 --sharding_config="default_shardings/llama.yaml"I am also able to start the benchmark script that sends grpc request with
jetstream-pytorch/deps/JetStream# python benchmarks/benchmark_serving.py --tokenizer $tokenizer_path --num-prompts 2000 --dataset-path $dataset_path --dataset sharegpt --save-request-outputsFor the server start I have to manually add --sharding_config="default_shardings/llama.yaml" otherwise it will complain it can't find sharding_config
for the benchmark script I found that --warm-up=True is no longer supported so I have to remove that option. Benchmark script eventually output.
Successful requests: 2000
Benchmark duration: 511.800847 s
Total input tokens: 220485
Total generated tokens: 622832
Request throughput: 3.91 requests/s
Input token throughput: 430.80 tokens/s
Output token throughput: 1216.94 tokens/s
Mean TTFT: 361578.43 ms
Median TTFT: 361889.06 ms
P99 TTFT: 493906.13 ms
Mean TPOT: 6019.10 ms
Median TPOT: 1346.14 ms
P99 TPOT: 111032.06 msI don't know if I can trust this data because there is a long warm up time in the beginning. The throughput also seems a bit low since I am using v4-8 with llama2-7b.
JackCaoG
commented
3 months ago Ah Thanks Han, the new flag is warmup-first instead of warm-up. With that I see the throuput of
Successful requests: 2000
Benchmark duration: 279.988127 s
Total input tokens: 220485
Total generated tokens: 623221
Request throughput: 7.14 requests/s
Input token throughput: 787.48 tokens/s
Output token throughput: 2225.88 tokens/s
Mean TTFT: 130993.14 ms
Median TTFT: 132435.39 ms
P99 TTFT: 262211.08 ms
Mean TPOT: 2297.64 ms
Median TPOT: 479.33 ms
P99 TPOT: 43248.99 msLet me try to submit a pr to fix the README and then we can close this issue.
I am trying to run benchmark on TPU v4-8. I followed the instructions on the README and download the weight and tokenizer for the Llama2.
I first try to do local run with
but it failed with
too many positional arguments.I then tried to do a benchmark run with
python benchmarks/benchmark_serving.py --tokenizer $tokenizer_path --num-prompts 2000 --dataset-path $dataset_path --dataset sharegpt --save-request-outputs --warm-up=True. It seems likewram_upflag is not longer thereafter I remove it, the new failure is
I then try to start the server with
now it failed with
I am wondering what did I do wrong here.