Issue when working with Azure and OpenAI Callback

Yanni8 commented 11 months ago

System Info

Langchain: v0.0.350 OS: Linux

Who can help?

@agola11

Information

[ ] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[X] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[X] Callbacks/Tracing
[ ] Async

Reproduction

The problem occurs when you use Azure with an GPT 4 Model because the Azure API will always respond with gpt-4 as the Model name. You can also see this in the official Microsoft documentation. https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision#output. It will therefore calculate the wrong price → if you use Turbo will the price will be x3 as it actually should be.

Code to Reproduce:


llm = AzureChatOpenAI(
     deployment_name="GPT4-TURBO"
)

with get_openai_callback() as cb:
    # Run LLM
    print((cb.total_tokens / 1000) * 0.01, "is instead", cb.total_cost)

Expected behavior

It should return the correct price.

dosubot[bot] commented 11 months ago

🤖

Hey there, @Yanni8! Good to see you again. Always a pleasure to cross paths with familiar faces. Let's dive into this issue you're encountering.

Based on the information you've provided, it seems like the issue is related to the model_name attribute in the BaseOpenAI class. By default, this attribute is set to "gpt-3.5-turbo-instruct". If you're using the GPT-4 model with Azure API, you should ensure that you're setting the model_name attribute to "gpt-4" when creating the BaseOpenAI instance. Here's an example of how you can do this:

openai_instance = BaseOpenAI(model_name="gpt-4", ...)

Replace ... with other parameters you need to pass to the BaseOpenAI constructor.

This should ensure that the correct model name is used, and the price calculation should be correct. You can find more details about this in the LangChain codebase.

Additionally, there have been similar issues reported in the LangChain repository. For instance, this issue was resolved by including the model parameter in the AzureChatOpenAI class initialization. This issue was resolved by adding the new models to the MODEL_COST_PER_1K_TOKENS dictionary in the openai_info.py file with their respective costs per 1,000 tokens.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best, Dosu

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

JulietteV commented 10 months ago

@dosu-bot, it seems that the cost of Azure gpt-35-turbo version 1106 is missing from the openai_info.py file.

# Azure GPT-35 input
"gpt-35-turbo": 0.0015,  # Azure OpenAI version of ChatGPT
"gpt-35-turbo-0301": 0.0015,  # Azure OpenAI version of ChatGPT
"gpt-35-turbo-0613": 0.0015,
"gpt-35-turbo-instruct": 0.0015,
"gpt-35-turbo-16k": 0.003,
"gpt-35-turbo-16k-0613": 0.003,
# Azure GPT-35 output
"gpt-35-turbo-completion": 0.002,  # Azure OpenAI version of ChatGPT
"gpt-35-turbo-0301-completion": 0.002,  # Azure OpenAI version of ChatGPT
"gpt-35-turbo-0613-completion": 0.002,
"gpt-35-turbo-instruct-completion": 0.002,
"gpt-35-turbo-16k-completion": 0.004,
"gpt-35-turbo-16k-0613-completion": 0.004,

Yanni8 commented 9 months ago

@hwchase17 any update on this issue?

spacepirate0001 commented 5 months ago

Having the same issue with Meta-Llama-3-70B-Instruct! {"cost":0.0,"total_tokens":0}

langchain-ai / langchain