NEW vLLM Update leads to attribute error when integrating

[ivysdad]

DEFAULT_MAX_NEW_TOKENS = 256

ListOrStrList = Union[str, List[str]]

tokenizer_data = build_vllm_token_enforcer_tokenizer_data(llm)

def vllm_with_character_level_parser(prompt: ListOrStrList, parser: Optional[CharacterLevelParser] = None) -> ListOrStrList:

    sampling_params = SamplingParams()
    sampling_params.max_tokens = DEFAULT_MAX_NEW_TOKENS
    if parser:
        logits_processor = build_vllm_logits_processor(tokenizer_data, parser)
        sampling_params.logits_processors = [logits_processor]

    results = llm.generate(prompt, sampling_params=sampling_params)
    if isinstance(prompt, str):
        return results[0].outputs[0].text
    else:
        return [result.outputs[0].text for result in results]

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [8], line 7
      3 DEFAULT_MAX_NEW_TOKENS = 256
      5 ListOrStrList = Union[str, List[str]]
----> 7 tokenizer_data = build_vllm_token_enforcer_tokenizer_data(llm)
      9 def vllm_with_character_level_parser(prompt: ListOrStrList, parser: Optional[CharacterLevelParser] = None) -> ListOrStrList:
     11     sampling_params = SamplingParams()

File /usr/local/lib/python3.9/dist-packages/lmformatenforcer/integrations/vllm.py:36, in build_vllm_token_enforcer_tokenizer_data(llm)
     34 def build_vllm_token_enforcer_tokenizer_data(llm: Union[vllm.LLM, PreTrainedTokenizerBase]) -> TokenEnforcerTokenizerData:
     35     tokenizer = llm.get_tokenizer() if isinstance(llm, vllm.LLM) else llm
---> 36     return build_token_enforcer_tokenizer_data(tokenizer)

File /usr/local/lib/python3.9/dist-packages/lmformatenforcer/integrations/transformers.py:70, in build_token_enforcer_tokenizer_data(tokenizer)
     69 def build_token_enforcer_tokenizer_data(tokenizer: PreTrainedTokenizerBase) -> TokenEnforcerTokenizerData:
---> 70     regular_tokens = _build_regular_tokens_list(tokenizer)
     71     def decode_fn(tokens: List[int]) -> str:
     72         decoded = tokenizer.decode(tokens)

File /usr/local/lib/python3.9/dist-packages/lmformatenforcer/integrations/transformers.py:58, in _build_regular_tokens_list(tokenizer)
     56 token_0 = tokenizer.encode("0")[-1]
     57 regular_tokens = []
---> 58 for token_idx in range(tokenizer.vocab_size):
     59     if token_idx in tokenizer.all_special_ids:
     60         continue

AttributeError: 'TokenizerGroup' object has no attribute 'vocab_size'

[noamgat]

Thanks for the report! Can you perhaps post the relevant vLLM release notes that talk about the API change so I can update the integration?

On Thu, Feb 1, 2024 at 10:45 AM Michael Nguyen @.***> wrote:

DEFAULT_MAX_NEW_TOKENS = 256

ListOrStrList = Union[str, List[str]]

tokenizer_data = build_vllm_token_enforcer_tokenizer_data(llm)

def vllm_with_character_level_parser(prompt: ListOrStrList, parser: Optional[CharacterLevelParser] = None) -> ListOrStrList:

sampling_params = SamplingParams()
sampling_params.max_tokens = DEFAULT_MAX_NEW_TOKENS
if parser:
    logits_processor = build_vllm_logits_processor(tokenizer_data, parser)
    sampling_params.logits_processors = [logits_processor]

results = llm.generate(prompt, sampling_params=sampling_params)
if isinstance(prompt, str):
    return results[0].outputs[0].text
else:
    return [result.outputs[0].text for result in results]

---

AttributeError Traceback (most recent call last) Cell In [8], line 7 3 DEFAULT_MAX_NEW_TOKENS = 256 5 ListOrStrList = Union[str, List[str]] ----> 7 tokenizer_data = build_vllm_token_enforcer_tokenizer_data(llm) 9 def vllm_with_character_level_parser(prompt: ListOrStrList, parser: Optional[CharacterLevelParser] = None) -> ListOrStrList: 11 sampling_params = SamplingParams()

File /usr/local/lib/python3.9/dist-packages/lmformatenforcer/integrations/vllm.py:36, in build_vllm_token_enforcer_tokenizer_data(llm) 34 def build_vllm_token_enforcer_tokenizer_data(llm: Union[vllm.LLM, PreTrainedTokenizerBase]) -> TokenEnforcerTokenizerData: 35 tokenizer = llm.get_tokenizer() if isinstance(llm, vllm.LLM) else llm ---> 36 return build_token_enforcer_tokenizer_data(tokenizer)

File /usr/local/lib/python3.9/dist-packages/lmformatenforcer/integrations/transformers.py:70, in build_token_enforcer_tokenizer_data(tokenizer) 69 def build_token_enforcer_tokenizer_data(tokenizer: PreTrainedTokenizerBase) -> TokenEnforcerTokenizerData: ---> 70 regular_tokens = _build_regular_tokens_list(tokenizer) 71 def decode_fn(tokens: List[int]) -> str: 72 decoded = tokenizer.decode(tokens)

File /usr/local/lib/python3.9/dist-packages/lmformatenforcer/integrations/transformers.py:58, in _build_regular_tokens_list(tokenizer) 56 token_0 = tokenizer.encode("0")[-1] 57 regular_tokens = [] ---> 58 for token_idx in range(tokenizer.vocab_size): 59 if token_idx in tokenizer.all_special_ids: 60 continue

AttributeError: 'TokenizerGroup' object has no attribute 'vocab_size'

— Reply to this email directly, view it on GitHub https://github.com/noamgat/lm-format-enforcer/issues/67, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKFA2F5BPNDEWHTQ4XV3PLYRNI3DAVCNFSM6AAAAABCUPG62WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYTCOBZHEYDOOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ivysdad]

Thanks for being so quick! Unfortunately I can't tell you where exactly is the issue (honestly not technical enough to understand what happened) but here is the release notes! my best guess would it having to do something with refactor api changes (this was one of the few API changes that seemed like it would change things) but here is the full change log. sorry I wasn't more helpful with this :(

[noamgat]

Solved in v0.8.3