[Usage]: Best engine arguments for large batch inference

[alpayariyak]

Your current environment

irrelevant

How would you like to use vllm

What would be the arguments that would maximize overall throughput for large batch offline inference? More specifically, I'm looking to generate 405B FP8 outputs for millions of inputs with 8x80 H100 SXM.

Thus far, I've been using the following arguments, but I womder if there are any others that would optimize this usecase, where per-request throughput and TTFT don't matter?

    --max-model-len 8192 \
    --disable-log-requests \
    --gpu_memory_utilization 0.93 \
    --use-v2-block-manager \
    --block-size 32 \
    --max-num-seqs 512

Before submitting a new issue...

• [X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[dipatidar]

@alpayariyak, I experimented with vllm config params for 405B FP8 on 8x80 H100 SXM. Setting --tensor-parallel-size 8 leverages all eight GPUs effectively, significantly boosting model throughput through parallel computation. does you data contain large input length/tokens? If so you can increase the --max-model-len 8192 . I was able to get 65k context length with --tensor-parallel-size 8 and by keeping other params to default values.

[alpayariyak]

@dipatidar thank you, but I'm already doing tp=8 and my max model len is adequate. The other parameters are what I'm asking about, as default values often try to balance per-request performance and ttft with overall throughput, and I only care about the latter.