-
### 🚀 The feature, motivation and pitch
This paper might be of interest: https://arxiv.org/pdf/2306.14048.pdf
This paper mentions removing a small portion of the KV cache does not affect the resul…
-
你好,PyramidKV 这个实现确实很巧妙,但是有个问题:我看了下 StreamingLLM 的源码,这个实现中,由于位置编码重新排列,所以注意力实现那里在存 kv cache 之前,并没有对 k 进行位置嵌入,在存 kv cache 后才对 k 进行了临时的位置嵌入和注意力计算;但是 PyramidKV 对应的 StreamingLLM 实现,是在对 q 和 k 同时进行位置嵌入之后进行了kv…
-
I run the example.py with llama2-7B-hf,set input length 4096 tokens,and output length 100 tokens. config.k_bits = 2, config.v_bits = 2. the kv cache occupy 5.6GB memory,only save about 500MB compared…
-
### Describe the issue
Thank you for the amazing work!
1. Does the model store the whole kv-cache of prefilling and generation on device? If so, how can the device hold the memory of 1M kv value…
-
Hello!
I have been really excited about your work! I attempted to use Palu for model compression on the Qwen2 series models, but regardless of the compression rate I set, I seem to encounter signif…
-
From our README.md
> torchao is a library to create and integrate high-performance custom data types layouts into your PyTorch workflows
And so far we've done a good job building out the primiti…
-
I am trying to use OpenAI GPT-2 model for prompt compression. However, getting error "OpenAIGPTLMHeadModel.forward() got an unexpected keyword argument 'past_key_values'". Has anyone faced/facing simi…
-
### 🚀 The feature, motivation and pitch
Allow saving of kv cache for faster TTFT on long prompts.
Another related feature would be to allow a fixed and shared cache across all requests. Perhaps th…
-
**Software Versions**
* Python: v3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)]
* OS: Windows10
* Kivy :v2.0.0
* Kivy installation method: pip installed in a venv u…
-
After attempting to upgrade from an older version of RocksDB to a newer one that includes #5298 and #5504, I started seeing a very significant slowdown in a point-lookup heavy workload. Profiling ind…