add structured prompt compress

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

https://llmlingua.com/

MIT License

4.27k stars 228 forks source link

add structured prompt compress #81

Closed SiyunZhao closed 5 months ago

SiyunZhao commented 5 months ago

Add structured_compress_prompt(), enabling users to define whether a section should be compressed and the degree of compression to apply. This allows for varied compression strategies based on full context, enhancing the compressor's efficiency and adaptability.

iofu728 commented 5 months ago

[ ] Add related documents;

iofu728 commented 5 months ago

Before commit, please run the make style.