[Question]: When I changed the target token, the code reported an error

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

https://llmlingua.com/

MIT License

4.18k stars 222 forks source link

[Question]: When I changed the target token, the code reported an error #135

Open macheng6 opened 2 months ago

macheng6 commented 2 months ago

Describe the issue

In the LLMLingua project, I attempted to use the qwen model instead of modelname and oai_tokenzier in the code, when the targte token is 150, the code runs normally. But when I changed this parameter, the code reported an error: The expanded size of the tensor (270) must match the existing size (298) at non-singleton dimension 0. Target sizes: [270]. Tensor sizes: [298] I tried to change the target token parameter, but still reported an error.

iofu728 commented 2 months ago

Hi @macheng6, thanks for your support.

We'll address this issue. Could you provide more details, such as the specific prompt and compressor model you used? This information will help us to reproduce and resolve the issue more effectively.