Support for OpenAI quotas

lefta commented 10 months ago

System Info

Python 3.11 Lanchain 315

Who can help?

@hwchase17 @agola11

Information

[X] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[X] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction

import openai import os from langchain.llms import OpenAI llm = OpenAI(temperature=0.3) print(llm.predict("What is the capital of India?"))

Expected behavior

When OpenAI quotas are reached (or no payment method is defined), requests should not be retried but should raise an appropriate error.

Error in console :

Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: You exceeded your current quota, please check your plan and billing details..

Equivalent TS issue : https://github.com/langchain-ai/langchainjs/issues/1929 Fix in TS land : https://github.com/langchain-ai/langchainjs/pull/1934

dosubot[bot] commented 10 months ago

🤖

Hello,

Thank you for bringing this to our attention. I understand that you're experiencing issues with the OpenAI API rate limit and the retry behavior of LangChain.

Currently, when the OpenAI API rate limit is reached, LangChain is designed to retry the request. This is handled by the _create_retry_decorator function, which creates a retry decorator that will be used to retry the API call when certain types of errors occur, including openai.error.RateLimitError. The maximum number of retries is determined by the max_retries attribute of the llm object, which defaults to 6.

Here is the relevant code:

def _create_retry_decorator(
    llm: Union[BaseOpenAI, OpenAIChat],
    run_manager: Optional[
        Union[AsyncCallbackManagerForLLMRun, CallbackManagerForLLMRun]
    ] = None,
) -> Callable[[Any], Any]:
    import openai

    errors = [
        openai.error.Timeout,
        openai.error.APIError,
        openai.error.APIConnectionError,
        openai.error.RateLimitError,
        openai.error.ServiceUnavailableError,
    ]
    return create_base_retry_decorator(
        error_types=errors, max_retries=llm.max_retries, run_manager=run_manager
    )

You can find this in the libs/langchain/langchain/llms/openai.py file.

However, I understand that you would prefer for an error to be raised immediately when the rate limit is reached, rather than retrying the request. This is a valid concern, and I believe it would be beneficial to have this as a configurable option in LangChain.

I will raise this issue with the LangChain development team for consideration in a future update. In the meantime, you could potentially modify the _create_retry_decorator function in your local copy of LangChain to remove openai.error.RateLimitError from the list of errors that trigger a retry, if this suits your needs.

I hope this helps, and please let me know if you have any further questions or concerns.

Best, Dosu

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

dosubot[bot] commented 7 months ago

Hi, @lefta,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, you raised the issue regarding adding support for OpenAI quotas in the Langchain Python library, specifically to handle retries and errors appropriately. I acknowledged the concern and suggested a potential workaround by modifying the _create_retry_decorator function locally. The issue is currently unresolved, and it seems like it needs further consideration from the LangChain development team for a future update.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

langchain-ai / langchain