openai / evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Other
14.35k stars 2.54k forks source link

Unified create_retrying for all solvers #1501

Closed ojaffe closed 3 months ago

ojaffe commented 3 months ago

We're now implementing solvers for new APIs we're calling (Anthropic, Gemini, ...). Each solver was implementing the same logic for backing off and retrying when the API query limit was hit. This PR created a generic create_retrying function, which retries when specific exceptions are raised. These exceptions are passed as arguments.

This uses the changes from #1482