search

zilliztech / GPTCache

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
https://gptcache.readthedocs.io
MIT License
6.96k stars 490 forks source link

[Feature]: Advanced OpenAI error handling #517

Open alexf-a opened 11 months ago

alexf-a commented 11 months ago

Is your feature request related to a problem? Please describe.

Bulk requests to OpenAI become difficult because of errors thrown by OpenAI due to rate limits. Errors that do not directly reflect a user's rate limit are also sometimes thrown.

If a script loops over a large amount of data and makes one request per data point, the script can crash and existing responses will be lost (costing time and money).

The two errors that I have regularly encountered are openai.error.RateLimitError and openai.error.APIConnectionError.

Describe the solution you'd like.

Automatic retries in response to openai errors. Client code should not worry at all about retry logic (timing, error handling, etc.).

Describe an alternate solution.

Cache existing responses, so that subsequent requests pick up where the script crashed.

Anything else? (Additional Context)

No response

SimFG commented 11 months ago

@alexf-a Thanks your suggestion, and i will try to improvement