search

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.18k stars 222 forks source link

Meaningless tokens generation #149

Open aravindpai opened 2 months ago

aravindpai commented 2 months ago

Describe the bug

Long LLM Lingua is generating the meaningless tokens and combining it.

For example, Fuel Dilution is combined to "uedil"

Why is it happening?

Steps to reproduce

No response

Expected Behavior

No response

Logs

No response

Additional Information

No response

iofu728 commented 2 months ago

Hi @aravindpai, thanks for your support. Yes, since LLMLingua performs token-level compression, it combines tokens into a new word. If you need to handle specific sensitive words, such as personal or geographical names, you can use the recovery function to restore the relevant content.

aravindpai commented 2 months ago

Hi @iofu728 What is a recovery function? and how do i restore the relevant content?

iofu728 commented 2 months ago

Hi @aravindpai, you can refer this document https://github.com/microsoft/LLMLingua/blob/main/DOCUMENT.md#post-processing to use recover function.