[Usage]: Seeing perf regression using chunked_prefill on VLLM 0.5.4

[jiahanc]

Your current environment

Docker container following build_from _source instruction

How would you like to use vllm

I want to experiment how chunked_prefill can increase throughput but see perf regression when enabling chunked_prefill. I am testing on Llama-3-70B-Instruct-Gradient-1048k on 8 H100 SXM GPUs. I am benchmarking using /benchmark/benchmark_throughput.py and Tensor Parallel size 8 Is there any guidance how to choose max_batched_tokens properly? My command is :

python3 ./benchmarks/benchmark_throughput.py \
    --model hf_model_path \
    --tokenizer hf_model_path \
    --tensor-parallel-size 8 \
    --num-prompts 50 \
    --input-len 40000 \
    --output-len 256 \
    --enable-chunked-prefill \
    --max_batched_tokens 4096
    --trust-remote-code 

My benchmark result is:

[KuntaiDu]

The rationale of picking max_batched_tokens: make it as large as possible, as long as your inter-token latency permits.

[KuntaiDu]

Also, chunked prefill does not improve offline throughput. The main goal of chunked prefill is:

1. improve throughput w.r.t. a latency target
2. reduce peak memory usage when handling large context

[jiahanc]

Thank you @KuntaiDu , will experiment with larger max_batched_tokens length

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!