fix: check against num_tokens instead of num_prompt_tokens for shared blocks

vectorch-ai / ScaleLLM

A high-performance inference system for large language models, designed for production environments.

https://docs.vectorch.com/

Apache License 2.0

377 stars 28 forks source link

Closed guocuimi closed 3 months ago

guocuimi commented 3 months ago

It is possible that shared_tokens > num_prompt_tokens for preempted sequence.