To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
I use Qwen-7B ,then get error:
Traceback (most recent call last):
File "/qwen/test.py", line 22, in
compressed_prompt = llm_lingua.compress_prompt(
File "/usr/local/lib/python3.10/dist-packages/llmlingua/prompt_compressor.py", line 253, in compress_prompt
context = self.iterative_compress_prompt(
File "/usr/local/lib/python3.10/dist-packages/llmlingua/prompt_compressor.py", line 749, in iterative_compress_prompt
past_loss[ready_end : end - 1] = loss
RuntimeError: The expanded size of the tensor (181) must match the existing size (211) at non-singleton dimension 0. Target sizes: [181]. Tensor sizes: [211]
I use Qwen-7B ,then get error: Traceback (most recent call last): File "/qwen/test.py", line 22, in
compressed_prompt = llm_lingua.compress_prompt(
File "/usr/local/lib/python3.10/dist-packages/llmlingua/prompt_compressor.py", line 253, in compress_prompt
context = self.iterative_compress_prompt(
File "/usr/local/lib/python3.10/dist-packages/llmlingua/prompt_compressor.py", line 749, in iterative_compress_prompt
past_loss[ready_end : end - 1] = loss
RuntimeError: The expanded size of the tensor (181) must match the existing size (211) at non-singleton dimension 0. Target sizes: [181]. Tensor sizes: [211]