search

langchain-ai / langchain-google

MIT License
108 stars 136 forks source link

ChatAnthropicVertex: get_num_tokens not working #372

Open IngLP opened 3 months ago

IngLP commented 3 months ago

Try to instantiate ChatAnthropicVertex and call get_num_tokens, it will fail:

  File "/Users/lorenzop/Library/Caches/pypoetry/virtualenvs/chatbotpresentazioni-EjUf89iu-py3.10/lib/python3.10/site-packages/langchain/memory/summary_buffer.py", line 66, in prune
    curr_buffer_length = self.llm.get_num_tokens_from_messages(buffer)
  File "/Users/lorenzop/Library/Caches/pypoetry/virtualenvs/chatbotpresentazioni-EjUf89iu-py3.10/lib/python3.10/site-packages/langchain_core/language_models/base.py", line 359, in get_num_tokens_from_messages
    return sum([self.get_num_tokens(get_buffer_string([m])) for m in messages])
  File "/Users/lorenzop/Library/Caches/pypoetry/virtualenvs/chatbotpresentazioni-EjUf89iu-py3.10/lib/python3.10/site-packages/langchain_core/language_models/base.py", line 359, in <listcomp>
    return sum([self.get_num_tokens(get_buffer_string([m])) for m in messages])
  File "/Users/lorenzop/Library/Caches/pypoetry/virtualenvs/chatbotpresentazioni-EjUf89iu-py3.10/lib/python3.10/site-packages/langchain_google_vertexai/_base.py", line 281, in get_num_tokens
    result = self.client_preview.count_tokens([text])
AttributeError: 'NoneType' object has no attribute 'count_tokens'

I am using langchain-google-vertexai = "^1.0.6"

lkuligin commented 3 months ago

I don't think Anthropic supports this: https://docs.anthropic.com/en/api/messages

IngLP commented 3 months ago

What is the best way to have an estimate? BTW: this breaks all modules and functions in LangChain that rely on get_num_tokens. I want to use a cache but I can't.

lkuligin commented 1 month ago

we can add a get_num_tokens method on the class level that would just give an estimation based on heuristics, wdyt?