search

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.42k stars 241 forks source link

[Bug]: NotImplementedError #168

Open isAXuan opened 2 months ago

isAXuan commented 2 months ago

Describe the bug

llmlingua-2 model is downed in hugging facing

code:

        self.llm_lingua = PromptCompressor( 
            model_name="/home/webservice/llm/compressFromNet/llmlingua-2-xlm",
            use_llmlingua2=True, # Whether to use llmlingua-2
            )

        compressed_prompt = self.llm_lingua.compress_prompt(
            context, # context  is str type
            rate=0.33, 
            force_tokens = ['\n', '?'],
            drop_consecutive=True,)
        print(compressed_prompt)

err:

      compressed_prompt = self.llm_lingua.compress_prompt(
  File "/home/webservice/miniconda3/envs/tka/lib/python3.10/site-packages/llmlingua/prompt_compressor.py", line 472, in compress_prompt
    return self.compress_prompt_llmlingua2(
  File "/home/webservice/miniconda3/envs/tka/lib/python3.10/site-packages/llmlingua/prompt_compressor.py", line 776, in compress_prompt_llmlingua2
    context_probs, context_words = self.__get_context_prob(
  File "/home/webservice/miniconda3/envs/tka/lib/python3.10/site-packages/llmlingua/prompt_compressor.py", line 2134, in __get_context_prob
    ) = self.__merge_token_to_word(
  File "/home/webservice/miniconda3/envs/tka/lib/python3.10/site-packages/llmlingua/prompt_compressor.py", line 2199, in __merge_token_to_word
    elif is_begin_of_new_word(token, self.model_name, force_tokens, token_map):
  File "/home/webservice/miniconda3/envs/tka/lib/python3.10/site-packages/llmlingua/utils.py", line 94, in is_begin_of_new_word
    raise NotImplementedError()
NotImplementedError

Steps to reproduce

No response

Expected Behavior

No response

Logs

No response

Additional Information

No response

iofu728 commented 1 month ago

Hi @isAXuan, thanks for your feedback.

It looks like the local model issue. Could you try to set model_name=microsoft/llmlingua-2-xlm-roberta-large-meetingbank for now?

jiapingW commented 2 weeks ago

Thank you very much for your reply. My problem has been solved. I checked my environment and it should be caused by the naming of the local model weight file.