search

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.42k stars 241 forks source link

RuntimeError: The expanded size of the tensor (181) must match the existing size (211) at non-singleton dimension 0 #36

Open kofuya opened 8 months ago

kofuya commented 8 months ago

I use Qwen-7B ,then get error: Traceback (most recent call last): File "/qwen/test.py", line 22, in compressed_prompt = llm_lingua.compress_prompt( File "/usr/local/lib/python3.10/dist-packages/llmlingua/prompt_compressor.py", line 253, in compress_prompt context = self.iterative_compress_prompt( File "/usr/local/lib/python3.10/dist-packages/llmlingua/prompt_compressor.py", line 749, in iterative_compress_prompt past_loss[ready_end : end - 1] = loss RuntimeError: The expanded size of the tensor (181) must match the existing size (211) at non-singleton dimension 0. Target sizes: [181]. Tensor sizes: [211]

iofu728 commented 8 months ago

Hi @kofuya,

Thanks for your support in our project. Could you give me more context, like the original prompt.