get_openai_callback() does not show the cost taken from new OpenAI's model ("gpt-4-1106-preview" and "gpt-3.5-turbo-1106")

[CLRafaelR]

When we employ the new models introduced during DevDay, gpt-3.5-turbo-1106 and gpt-4-1106-preview, get_openai_callback() does not accurately display their total cost, which includes both the prompt and completion token cost. The subsequent minimal working example illustrates this issue. The costs should not be $0.0, but rather $0.01 30 + $0.03 598 = $18.24 for gpt-4-1106-preview and $0.0010 30 + $0.0020 160 = $0.35 for gpt-3.5-turbo-1106.

MWE

from langchain.schema.messages import HumanMessage, SystemMessage
from langchain.chat_models import ChatOpenAI
from langchain.callbacks import get_openai_callback
# export OPENAI_API_KEY="..."

replies = []
for model in [
    "gpt-4-1106-preview",
    "gpt-3.5-turbo-1106",
]:
    chat = ChatOpenAI(model=model)
    messages = [
        SystemMessage(content="You're a helpful assistant"),
        HumanMessage(content="What is the purpose of model regularization? Explain comprehensibly in English."),
    ]
    with get_openai_callback() as cb:
        reply = chat.invoke(messages).content
    print(cb, reply, sep="\n------\n")
    print("="*16)
    replies.append(reply)

replies

Output of MWE

Tokens Used: 628
    Prompt Tokens: 30
    Completion Tokens: 598
Successful Requests: 1
Total Cost (USD): $0.0
------
Model regularization is a technique used in machine learning to prevent a model from overfitting the training data and to improve its generalization to unseen data. Overfitting occurs when a model learns the training data too well, including its noise and outliers, which often results in poor performance on new, unseen data because the model is too tailored to the specifics of the training set.

Here is a comprehensible explanation of the purpose of regularization:

1. **Prevent Overfitting**: Regularization helps to keep the model simple by introducing a penalty for more complex models. By doing so, it discourages the learning of a model that is too complex for the underlying pattern in the data. A simpler model may not perform as well on the training data, but it can perform better on new data because it captures the general trend rather than the specific noise.

2. **Improves Generalization**: The main goal of a machine learning model is to make accurate predictions on new, unseen data. Regularization helps by ensuring that the model's performance during training is more reflective of how it will perform on new data. This is achieved by penalizing the complexity of the model and thereby encouraging the model to be more robust to variations in data.

3. **Controls Model Complexity**: Regularization techniques typically introduce a hyperparameter that controls the trade-off between fitting the training data well and keeping the model complexity low. By adjusting this hyperparameter, one can find a good balance where the model is complex enough to capture the underlying patterns but not so complex that it starts fitting the noise.

4. **Introduces Bias to Reduce Variance**: In statistical terms, regularization introduces a small amount of bias to the model to significantly reduce its variance. This trade-off is beneficial because high variance models are those that overfit the data, while a little bias can make the model more stable and accurate in its predictions on new data.

5. **Handles Multicollinearity**: In cases where the features (input variables) are highly correlated, it can be difficult for the model to estimate the relationship of each feature with the output variable. Regularization techniques can help reduce the impact of multicollinearity by penalizing the coefficients of the correlated features, leading to more stable estimates.

Common regularization techniques include:

- **L1 Regularization (Lasso)**: This adds a penalty equal to the absolute value of the magnitude of coefficients. It can lead to some coefficients being shrunk to zero, effectively performing feature selection.
- **L2 Regularization (Ridge)**: This adds a penalty equal to the square of the magnitude of coefficients. This generally shrinks coefficients evenly but does not set them to zero.
- **Elastic Net**: This is a combination of L1 and L2 regularization and balances the properties of both methods.

In summary, regularization is a crucial step in building machine learning models that are effective at making predictions on new, unseen data by keeping the models simpler and more generalizable.
================
Tokens Used: 194
    Prompt Tokens: 30
    Completion Tokens: 164
Successful Requests: 1
Total Cost (USD): $0.0
------
Model regularization is a technique used in machine learning to prevent overfitting. Overfitting occurs when a model learns the training data too well and performs poorly on new, unseen data. Regularization helps to address this issue by adding a penalty to the model's learning process, discouraging it from becoming too complex and fitting noise in the data.

There are different types of regularization, such as L1 and L2 regularization, which add a penalty based on the magnitude of the model's coefficients. By using regularization, the model is encouraged to focus on the most important features in the data and avoid being overly sensitive to small fluctuations.

In simple terms, model regularization helps to keep the model in check and prevent it from becoming too specialized to the training data, improving its ability to make accurate predictions on new, unseen data.
================

It is necessary to include the cost per 1,000 tokens in MODEL_COST_PER_1K_TOKENS which is defined in the following lines, in accordance with OpenAI's official pricing page (here for gpt-4-turbo; here for gpt-3.5-turbo).

https://github.com/langchain-ai/langchain/blob/ff87f4b4f90c1d13ddb79120c6ded6c0af2959b7/libs/langchain/langchain/callbacks/openai_info.py#L7C1-L35

[dosubot[bot]]

🤖

Hello @CLRafaelR,

I'm Dosu, an AI bot here to assist you with your needs related to the LangChain repository. I can help you debug issues, answer your questions, and provide guidance on how to contribute to our project. While you're waiting for a human maintainer, I'm here to provide initial support.

I'm currently reviewing your issue and will get back to you with a more complete response shortly.

[dosubot[bot]]

🤖

Thank you for bringing this to our attention. It seems like the get_openai_callback() function is not currently set up to handle the newly introduced models gpt-3.5-turbo-1106 and gpt-4-1106-preview.

The get_openai_callback() function calculates the cost based on the MODEL_COST_PER_1K_TOKENS dictionary, which currently does not include the new models. This is why the cost is not being displayed correctly.

To resolve this issue, you would need to add the new models to the MODEL_COST_PER_1K_TOKENS dictionary in the openai_info.py file with their respective costs per 1,000 tokens. Here's how you can do it:

MODEL_COST_PER_1K_TOKENS = {
    ...
    "gpt-3.5-turbo-1106": <cost>,
    "gpt-3.5-turbo-1106-completion": <cost>,
    "gpt-4-1106-preview": <cost>,
    "gpt-4-1106-preview-completion": <cost>,
    ...
}

Please replace <cost> with the actual cost per 1,000 tokens for each model. Once you've done this, the get_openai_callback() function should be able to calculate the total cost for these models correctly.

I hope this helps! If you have any other questions or run into any issues, please let me know.

Sources

• Token usage calculation is not working for ChatOpenAI
• The cost calculation of tokens for open ai models looks like is only considering output tokens
• get_openai_callback total_cost BROKEN
• libs/langchain/langchain/callbacks/openai_info.py
• libs/langchain/langchain/callbacks/manager.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[CLRafaelR]

13244 will resolve the issue reported here.

[IsakNyberg]

On the topic of this I think the current way of reporting 0 for unknown models is not ideal. Especially since somebody might falsely be relying on the cost data and then be billed more than they expect. Instead i think that if the cost data cannot be calculated the callback should report an error instead of reporting back 0$.

[CLRafaelR]

@IsakNyberg I totally agree to you. An error message should appear.

[CLRafaelR]

13244 was successfully merged https://github.com/langchain-ai/langchain/pull/13244#event-10935213932 hence, I close this issue.

[huanvo88]

Can we also update the pricing information for the latest OpenAI models (released 0125)?