Add file system as the optional storage for KVTensor in vineyard llm kv cache

v6d-io / v6d

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

https://v6d.io

Apache License 2.0

819 stars 117 forks source link

Add file system as the optional storage for KVTensor in vineyard llm kv cache #1833

Closed dashanji closed 3 months ago

dashanji commented 4 months ago

Describe your problem

Currently, the KVTensor use the vineyard blob as the backend storage. To support the chunk matching of prompts, we can use the the file system as the backend storage for KVTensor as well.