Improve the query API of llm cache and use vector<uint8_t> as payload object. - Githubissues

v6d-io / v6d

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

https://v6d.io

Apache License 2.0

819 stars 117 forks source link

Improve the query API of llm cache and use vector<uint8_t> as payload object. #1797

Closed dashanji closed 4 months ago

dashanji commented 4 months ago

What do these changes do?

Improve the query API, users only input a token list and will get the kv_cache with the longest prefix.
Use vector as payload object.
Replace the alias of KV_STATE_WITH_LAYER with std::map<int, std::pair<K_STATE, V_STATE>>.
Rename the Dimension with TensorBytes.
Use the references of std::vector<T> to avoid copying.
Print the rax tree to a string for debugging.

Related issue number

Fixes #1786
Fixes #1792
Fixes #1795