-
Hi, I'm wondering if you have any plans regarding kv compression methods like SnapKV and PyramidKV. These methods can reduce the use of memory for KV cache, hence improving availability on low-memory …
-
Hello,
I see SnapKV is used for the Heavy Hitter Prompt Compression strategy. As far as I understand (correct me if I'm wrong), it is also used in the benchmarks reported in the blog post for the H…
-
### Motivation
For current large model inference, KV cache occupies a significant portion of GPU memory, so reducing the size of KV cache is an important direction for improvement. Recently, severa…
-
Solid idea and Ingenious code implementations, Great work!
Have you considered implementing KV Compression operations on KV Cache in the vLLM framework?
-
### Motivation.
There are more and more use cases, where we need to transfer KV caches between vLLM instances, or store KV caches for future use. Some concrete use cases:
- Disaggregated prefilling.…
-
https://github.com/exo-explore/exo/issues/23#issuecomment-2241521048
Perhaps after each inference, we synchronise the full kv cache between all nodes. This should be fairly straightforward, we can …
-
There seems to be a serious issue in `run_longbench.py`. The `update_kv` is only called during the first sample in `longbench`, therefore the statement `print(f"PyramidKV max_capacity_prompt {max_capa…
-
It would be great if MLX_lm supported a --cache_prompt flag like in llama.cpp's integration ([link to their discussion + eventual PR](https://github.com/ml-explore/mlx-examples/issues/new)).
This …
-
### 🚀 The feature, motivation and pitch
This paper might be of interest: https://arxiv.org/pdf/2306.14048.pdf
This paper mentions removing a small portion of the KV cache does not affect the resul…
-
你好,PyramidKV 这个实现确实很巧妙,但是有个问题:我看了下 StreamingLLM 的源码,这个实现中,由于位置编码重新排列,所以注意力实现那里在存 kv cache 之前,并没有对 k 进行位置嵌入,在存 kv cache 后才对 k 进行了临时的位置嵌入和注意力计算;但是 PyramidKV 对应的 StreamingLLM 实现,是在对 q 和 k 同时进行位置嵌入之后进行了kv…