Question about batching strategy

hao-ai-lab / LookaheadDecoding

[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

https://arxiv.org/abs/2402.02057

Apache License 2.0

1.15k stars 67 forks source link

Question about batching strategy #66

Open lxnlxnlxnlxnlxn opened 1 month ago

lxnlxnlxnlxnlxn commented 1 month ago

Hi, I'm wondering whether lookahead decoding is orthogonal with any batching strategies. Batch size is set to 1 in the evaluation part. Whether lookahead decoding framework supports a higher batch size, for example, 10 ?