search

microsoft / LLMLingua

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.69k stars 262 forks source link

[Bug]: ZeroDivisionError: float division by zero upon using compress_json with `use_llmlingua2` #183

Open r4881t opened 2 months ago

r4881t commented 2 months ago

Describe the bug

I am trying to use compress_json and encountering ZeroDivisionError

Steps to reproduce

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(
      model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
      use_llmlingua2=True,
      device_map="cpu",
    )

json_data = {
        "query": "This is a question for LLM 1",
        "answer": "This is an answer from LLM 1"
    }

json_config = {
        "query": {
          "rate": 1.0,
          "compress": False,
          "value_type": "str",
          "pair_remove": False
        },
        "answer": {
          "rate": 0.4,
          "compress": True,
          "value_type": "str",
          "pair_remove": False
        }
      }

compressed_data = llm_lingua.compress_json(json_data, json_config, use_keyvalue_level_filter=False)

Note the error.

Expected Behavior

The error should not come.

Logs

ZeroDivisionError                         Traceback (most recent call last)
Cell In[21], [line 6](vscode-notebook-cell:?execution_count=21&line=6)
      [1](vscode-notebook-cell:?execution_count=21&line=1) llm_lingua = PromptCompressor(
      [2](vscode-notebook-cell:?execution_count=21&line=2)       model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
      [3](vscode-notebook-cell:?execution_count=21&line=3)       use_llmlingua2=True,
      [4](vscode-notebook-cell:?execution_count=21&line=4)       device_map="cpu",
      [5](vscode-notebook-cell:?execution_count=21&line=5)     )
----> [6](vscode-notebook-cell:?execution_count=21&line=6) compressed_data = llm_lingua.compress_json(x[0], json_config, use_keyvalue_level_filter=False)

File ~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:242, in PromptCompressor.compress_json(self, json_data, json_config, instruction, question, rate, target_token, iterative_size, use_sentence_level_filter, use_keyvalue_level_filter, use_token_level_filter, keep_split, keep_first_sentence, keep_last_sentence, keep_sentence_number, high_priority_bonus, context_budget, token_budget_ratio, condition_in_question, reorder_keyvalue, condition_compare, rank_method)
    [240](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:240) print(f">> {context}")
    [241](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:241) print(f">> {force_context_ids}")
--> [242](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:242) compressed_res = self.structured_compress_prompt(
    [243](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:243)     context=context,
    [244](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:244)     instruction=instruction,
    [245](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:245)     question=question,
    [246](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:246)     rate=rate,
    [247](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:247)     target_token=target_token,
    [248](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:248)     iterative_size=iterative_size,
    [249](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:249)     force_context_ids=force_context_ids,
    [250](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:250)     use_sentence_level_filter=use_sentence_level_filter,
    [251](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:251)     use_context_level_filter=use_keyvalue_level_filter,
    [252](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:252)     use_token_level_filter=use_token_level_filter,
    [253](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:253)     keep_split=keep_split,
    [254](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:254)     keep_first_sentence=keep_first_sentence,
    [255](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:255)     keep_last_sentence=keep_last_sentence,
    [256](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:256)     keep_sentence_number=keep_sentence_number,
    [257](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:257)     high_priority_bonus=high_priority_bonus,
    [258](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:258)     context_budget=context_budget,
    [259](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:259)     token_budget_ratio=token_budget_ratio,
    [260](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:260)     condition_in_question=condition_in_question,
    [261](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:261)     reorder_context=reorder_keyvalue,
    [262](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:262)     condition_compare=condition_compare,
    [263](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:263)     add_instruction=False,
    [264](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:264)     rank_method=rank_method,
    [265](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:265)     concate_question=False,
    [266](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:266)     strict_preserve_uncompressed=False,
    [267](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:267) )
    [268](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:268) compressed_json_text = remove_consecutive_commas(
    [269](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:269)     compressed_res["compressed_prompt"]
    [270](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:270) )
    [271](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:271) compressed_res["compressed_prompt"] = json.loads(compressed_json_text)

File ~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:394, in PromptCompressor.structured_compress_prompt(self, context, instruction, question, rate, target_token, iterative_size, force_context_ids, force_context_number, use_sentence_level_filter, use_context_level_filter, use_token_level_filter, keep_split, keep_first_sentence, keep_last_sentence, keep_sentence_number, high_priority_bonus, context_budget, token_budget_ratio, condition_in_question, reorder_context, dynamic_context_compression_ratio, condition_compare, add_instruction, rank_method, concate_question, strict_preserve_uncompressed)
    [387](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:387)     rate = target_token / sum(context_tokens_length)
    [388](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:388) (
    [389](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:389)     context,
    [390](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:390)     context_segs,
    [391](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:391)     context_segs_rate,
    [392](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:392)     context_segs_compress,
    [393](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:393) ) = self.segment_structured_context(context, rate)
--> [394](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:394) return self.compress_prompt(
    [395](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:395)     context,
    [396](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:396)     instruction,
    [397](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:397)     question,
    [398](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:398)     rate,
    [399](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:399)     target_token,
    [400](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:400)     iterative_size,
    [401](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:401)     force_context_ids,
    [402](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:402)     force_context_number,
    [403](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:403)     use_sentence_level_filter,
    [404](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:404)     use_context_level_filter,
    [405](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:405)     use_token_level_filter,
    [406](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:406)     keep_split,
    [407](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:407)     keep_first_sentence,
    [408](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:408)     keep_last_sentence,
    [409](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:409)     keep_sentence_number,
    [410](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:410)     high_priority_bonus,
    [411](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:411)     context_budget,
    [412](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:412)     token_budget_ratio,
    [413](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:413)     condition_in_question,
    [414](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:414)     reorder_context,
    [415](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:415)     dynamic_context_compression_ratio,
    [416](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:416)     condition_compare,
    [417](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:417)     add_instruction,
    [418](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:418)     rank_method,
    [419](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:419)     concate_question,
    [420](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:420)     context_segs=context_segs,
    [421](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:421)     context_segs_rate=context_segs_rate,
    [422](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:422)     context_segs_compress=context_segs_compress,
    [423](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:423)     strict_preserve_uncompressed=strict_preserve_uncompressed,
    [424](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:424) )

File ~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:537, in PromptCompressor.compress_prompt(self, context, instruction, question, rate, target_token, iterative_size, force_context_ids, force_context_number, use_sentence_level_filter, use_context_level_filter, use_token_level_filter, keep_split, keep_first_sentence, keep_last_sentence, keep_sentence_number, high_priority_bonus, context_budget, token_budget_ratio, condition_in_question, reorder_context, dynamic_context_compression_ratio, condition_compare, add_instruction, rank_method, concate_question, context_segs, context_segs_rate, context_segs_compress, target_context, context_level_rate, context_level_target_token, return_word_label, word_sep, label_sep, token_to_word, force_tokens, force_reserve_digit, drop_consecutive, chunk_end_tokens, strict_preserve_uncompressed)
    [469](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:469) """
    [470](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:470) Compresses the given context.
    [471](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:471) 
   (...)
    [534](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:534)         - "saving" (str): Estimated savings in GPT-4 token usage.
    [535](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:535) """
    [536](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:536) if self.use_llmlingua2:
--> [537](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:537)     return self.compress_prompt_llmlingua2(
    [538](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:538)         context,
    [539](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:539)         rate=rate,
    [540](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:540)         target_token=target_token,
    [541](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:541)         use_context_level_filter=use_context_level_filter,
    [542](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:542)         use_token_level_filter=use_token_level_filter,
    [543](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:543)         target_context=target_context,
    [544](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:544)         context_level_rate=context_level_rate,
    [545](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:545)         context_level_target_token=context_level_target_token,
    [546](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:546)         force_context_ids=force_context_ids,
    [547](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:547)         return_word_label=return_word_label,
    [548](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:548)         word_sep=word_sep,
    [549](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:549)         label_sep=label_sep,
    [550](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:550)         token_to_word=token_to_word,
    [551](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:551)         force_tokens=force_tokens,
    [552](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:552)         force_reserve_digit=force_reserve_digit,
    [553](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:553)         drop_consecutive=drop_consecutive,
    [554](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:554)         chunk_end_tokens=chunk_end_tokens,
    [555](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:555)     )
    [556](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:556) assert (
    [557](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:557)     rate <= 1.0
    [558](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:558) ), "Error: 'rate' must not exceed 1.0. The value of 'rate' indicates compression rate and must be within the range [0, 1]."
    [560](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:560) if not context:

File ~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:928, in PromptCompressor.compress_prompt_llmlingua2(self, context, rate, target_token, use_context_level_filter, use_token_level_filter, target_context, context_level_rate, context_level_target_token, force_context_ids, return_word_label, word_sep, label_sep, token_to_word, force_tokens, force_reserve_digit, drop_consecutive, chunk_end_tokens)
    [925](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:925)     return res
    [927](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:927) if target_token > 0:
--> [928](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:928)     rate = min(target_token / n_original_token, 1.0)
    [930](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:930) if use_token_level_filter:
    [931](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:931)     compressed_context, word_list, word_label_list = self.__compress(
    [932](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:932)         context_chunked,
    [933](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:933)         reduce_rate=max(0, 1 - rate),
   (...)
    [938](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:938)         drop_consecutive=drop_consecutive,
    [939](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e11.vscode-resource.vscode-cdn.net/home/pranav/workspace/merokudao/friday/notebooks/~/workspace/merokudao/friday/venv/lib/python3.12/site-packages/llmlingua/prompt_compressor.py:939)     )

ZeroDivisionError: float division by zero

Additional Information

I patched the code locally to try to print and see if something went wrong, it seems like the tokens are empty strings

[{'query': 'This is a question for LLM 1', 'answer': 'This is an answer from LLM 1'}]
>>> Doing data {'query': 'This is a question for LLM 1', 'answer': 'This is an answer from LLM 1'}
>> ['<llmlingua, compress=False>{</llmlingua>', '<llmlingua, compress=False>"query": "This is a question for LLM 1", </llmlingua>', '<llmlingua, compress=False>"answer": "</llmlingua><llmlingua, rate=0.4>This is an answer from LLM 1</llmlingua><llmlingua, compress=False>"</llmlingua>', '<llmlingua, compress=False>}</llmlingua>']
>> [0, 1, 2, 3]
>> ['', '', '', '']
>> ['', '', '', '']
>> 0
>> ['', '', '', '']
>> 0
>> ['', '', '', '']
>> 0
>> ['', '', '', '']
>> 0
iofu728 commented 1 month ago

Hi @r4881t, thanks for your feedback. However, LLMLingua-2 currently does not support the compression_json mode, so there may be some bugs.