v6d-io / v6d

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)
https://v6d.io
Apache License 2.0
819 stars 117 forks source link

Delete the tensor builder from vineyard llm kv cache to reduce the dependencies like arrow #1818

Closed dashanji closed 4 months ago

dashanji commented 4 months ago

Describe your problem

In the current design of llm kv cache, we use the tensor builder to build the blob of kv cache. However, we don't use the arrow buffer of the tensor builder. It's better to delete the tensor builder dependencies and create a new class to wrapper the tensor builder without the arrow dependencies.

Original:

$ ldd libvineyard_llm_cache.so | grep arrow
libarrow.so.1500 => /lib/x86_64-linux-gnu/libarrow.so.1500 (0x00007f5da9879000)

Expected:

$ ldd libvineyard_llm_cache.so | grep arrow