search

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.42k stars 241 forks source link

force_context_ids parameter not behaving as expected #58

Closed Dakraid closed 7 months ago

Dakraid commented 7 months ago

Hello, I've recently created a small service that intercepts calls from my frontend to my LLM backend to compress the incoming prompts. To retain some crucial parts, I wanted to make use of the force_context_ids, but as it turns out, I'm getting an error I can't further investigate.

File "C:\Users\micro\miniconda3\envs\LLMLinguaMITM\Lib\site-packages\llmlingua\prompt_compressor.py", line 200, in compress_prompt context, dynamic_ratio = self.control_context_budget( ^^^^^^^^^^^^^^^^^^^^^^ ValueError: too many values to unpack (expected 2)

I use this method to search the list of strings for the context parameter for matches against the regex and add them to a list of integers that represent their position in that first list. The list of integers is then handed to the compress_prompt() call.

forced_context_ids = []

for i, string in enumerate(split_prompts):
    if re.match(config['force_context'], string):
        forced_context_ids.append(i)

The full code can be found here: https://github.com/Dakraid/LLMLinguaMITM/blob/main/main.py

During debugging, the list of integers seemed fine and matched the expected positions in the list of strings for the context. Any further help or insight would be appreciated.

iofu728 commented 7 months ago

Hi @Dakraid, I deeply appreciate your efforts and for pointing out the bug. This issue has now been fixed, and we will release a new pip package as soon as possible.

You can update to the latest version by using pip install git+https://github.com/microsoft/LLMLingua.git.