search

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.48k stars 251 forks source link

[Question]: How to combine longllmlingua and llmlingua2 ? #162

Open qminh369 opened 4 months ago

qminh369 commented 4 months ago

Describe the issue

No response

iofu728 commented 4 months ago

Hi @qminh369, thanks for your support in LLMLingua. Currently, the pip package does not support using LongLLMLingua and LLMLingua-2 together. However, you can use them separately:

  1. First, use LongLLMLingua for coarse-level compression. 
    use_token_level_filter=False,
  2. Then, use LLMLingua-2 for token-level compression.
use_context_level_filter=False,
use_token_level_filter=True,