-
### Checklist
- [x] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.…
-
### Motivation
For current large model inference, KV cache occupies a significant portion of GPU memory, so reducing the size of KV cache is an important direction for improvement. Recently, severa…
-
Hi, I'm wondering if you have any plans regarding kv compression methods like SnapKV and PyramidKV. These methods can reduce the use of memory for KV cache, hence improving availability on low-memory …
-
Hello,
I see SnapKV is used for the Heavy Hitter Prompt Compression strategy. As far as I understand (correct me if I'm wrong), it is also used in the benchmarks reported in the blog post for the H…
-
Solid idea and Ingenious code implementations, Great work!
Have you considered implementing KV Compression operations on KV Cache in the vLLM framework?
-
### Motivation.
There are more and more use cases, where we need to transfer KV caches between vLLM instances, or store KV caches for future use. Some concrete use cases:
- Disaggregated prefilling.…
-
### What happened?
When you enable self extend eventually model will stop working because of "ERR [ update_slots] failed to decode the batch: KV cache is full - try increasing it via the c…
-
https://github.com/exo-explore/exo/issues/23#issuecomment-2241521048
Perhaps after each inference, we synchronise the full kv cache between all nodes. This should be fairly straightforward, we can …
-
Hello,
I managed to convert the gpt3 pax ckpt using convert_gpt3_ckpt_from_paxml.py script with 4 nodes (8 gpus in each node) and the follwing base_args:
```
base_args = [
"",
"MaxT…
-
There seems to be a serious issue in `run_longbench.py`. The `update_kv` is only called during the first sample in `longbench`, therefore the statement `print(f"PyramidKV max_capacity_prompt {max_capa…