[Question]: Achieved compression rate with (Long)LLMLingua not meeting expectations?

[cornzz]

Describe the issue

I was evaluating how well the (Long)LLMLingua is able to achieve the requested compression rate (focusing on the rate parameter, not target_tokens) and came to these conclusions:

• For smaller prompts (< 150 tokens) barely any compression can be achieved, if any at all
• Requested compression rate is best achieved for prompts around 2000 tokens
• For longer prompts (>5000 tokens) the requested rate is overshot (or undershot)

More detailed results are below. My question is, am doing something wrong when invoking LLMLingua, or is this behaviour normal? I adhered to the usage examples in README.md:

Code snippet

```python compressor = PromptCompressor( model_name="NousResearch/Llama-2-7b-hf", # or "openai-community/gpt2" device_map="balanced" ) ... def compress(prompt, rate, question=""): if longllmlingua: res = compressor.compress_prompt( [prompt], question=question, rate=rate, condition_in_question="after_condition", reorder_context="sort", dynamic_context_compression_ratio=0.3, condition_compare=True, rank_method="longllmlingua", ) else: res = compressor.compress_prompt(prompt, rate=rate) return res ```

I tested with the default Llama 2 7b aswell as with GPT-2. It seems that with the smaller model the deviation overall is smaller than with the bigger model.

(Prompt lengths measured using the GPT-3.5 tokenizer)

LLMLingua with Llama 2


LLMLingua with GPT-2


LongLLMLingua with Llama 2


LongLLMLingua with GPT-2


In contrast, LLMLingua-2 adheres to the requested compression rate quite well, only slightly overshooting the requested rate:

LLMLingua-2


The prompts I used are truncated from the longest prompt in the LongBench GovReport task (link).

[cornzz]

-- Moved to separate issue: https://github.com/microsoft/LLMLingua/issues/196 --