It was created a TimeoutHandler to assing timeouts.
For LLMs it depends of 3 variables:
Prefill Time a.k.a Time-To-First-Token (TTFT)
Decode Time a.k.a Time-Per-Output-Token (TPOT)
Queue Time
Prefill time (TTFT)
It was modeled as a quadratic functions [1]. Besides (0, 0), two more points are nedded. This two point acts as a SLA. For insntance, this points are (8192,2) and (32768, 10).
Decode time (TPOT)
The decode time is modeled as a linear function with respect to the average silent reading time in English. According to the reference [2], the average silent reading speed is 238 words per minute. Given that there are approximately 0.75 tokens per word, the TPOT can be calculated as follows:
It was created a TimeoutHandler to assing timeouts. For LLMs it depends of 3 variables:
Prefill time (TTFT)
It was modeled as a quadratic functions [1]. Besides (0, 0), two more points are nedded. This two point acts as a SLA. For insntance, this points are (8192,2) and (32768, 10).
Decode time (TPOT)
The decode time is modeled as a linear function with respect to the average silent reading time in English. According to the reference [2], the average silent reading speed is 238 words per minute. Given that there are approximately 0.75 tokens per word, the TPOT can be calculated as follows:
$
TPOT ~(ms/tok) = \frac{1000 ms}{1seg *~\frac{238 ~words}{min}*\frac{ 0.75 ~tok}{word}*\frac{1min}{60seg} } = \frac{1000ms}{2.975tok} = 336 \frac{ms}{tok}$Queue time
The queue time has been defined as 30 seconds.
Close #75
[1] https://arxiv.org/abs/2407.07000 [2] https://www.sciencedirect.com/science/article/abs/pii/S0749596X19300786
Close #75