search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
13.39k stars 1.57k forks source link

[Bug]: token_counter seems broken #1825

Closed toniengelhardt closed 8 months ago

toniengelhardt commented 8 months ago

What happened?

This code returns 3, when it should be ~11.

It used to work, but broke with one of the last updates.

Screenshot 2024-02-05 at 16 05 21

Relevant log output

No response

Twitter / LinkedIn details

No response

toniengelhardt commented 8 months ago

This also breaks the completion_cost function.

krrishdholakia commented 8 months ago

I believe you're just missing the required flag count_response_tokens

def test_token_counter(): 
    from litellm import token_counter

    model = 'gpt-3.5-turbo'  # 'gpt-4-1106-preview'
    text = 'Hello, this is a text with roughly 11 tokens.'

    count = token_counter(model=model, text=text, count_response_tokens=True)

    print(count)
    raise Exception("it worked!")
krrishdholakia commented 8 months ago

also added this scenario in our token_counter logic

should be fixed in v1.22.4