[Runtime] Stateless interface of PagedKVCache leaf node commit

apache / tvm

Open deep learning compiler stack for cpu, gpu and specialized accelerators

https://tvm.apache.org/

Apache License 2.0

11.42k stars 3.4k forks source link

[Runtime] Stateless interface of PagedKVCache leaf node commit #17057

Closed MasterJH5574 closed 1 month ago

MasterJH5574 commented 1 month ago

This PR changes the interface of the function CommitAcceptedTokenTreeNodeToKVCache introduced recently for PagedKVCache to a stateless interface. Previously the interace is a stateful one, which makes strong assumption on the caller side. This commit removes the assumption so that the interface becomes less confusing.