search

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.27k stars 228 forks source link

Experiments with Alphanumeric Entities #93

Open jasonngap1 opened 5 months ago

jasonngap1 commented 5 months ago

Hi there! Thank you for the wonderful work done as this greatly reduced the memory overhead and increased inference time for my use case. I noticed that the prompt compression in some context that involves alphanumeric entities like car plate numbers and ID numbers causes them to be corrupted.

Are there any experiments that you have conducted with such entities or any optimisation techniques that can mitigate this issue that you can share please? I am currently using the prompt compression with default parameters.

iofu728 commented 5 months ago

Hi @jasonngap1,

Thank you for your interest in our method. We acknowledge the issue of losing sensitive entities and will release a new piece of work to address this problem. Currently, if users need to manually specify entities that must be preserved, you can find the relevant code in the develop branch.

jasonngap1 commented 5 months ago

Thank you for working on this issue. May I please know where exactly in the develop branch can I manually specify the entities to preserve?

iofu728 commented 4 months ago

Hi @jasonngap1, you can follow the scripts to use that.