AnthropicSolver - Githubissues

openai / evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Other

14.35k stars 2.54k forks source link

AnthropicSolver #1498

Closed thesofakillers closed 3 months ago

thesofakillers commented 3 months ago

This PR contributes an AnthropicSolver class, a solver for using models served by the Anthropic Claude API, such as claude 3.

Besides basic functionality, the solver provides the following features

[x] Handles backoff
[x] Handles CoT and other solvers with non-alternating roles
[x] token usage estimate

Notes:

logit biasing not supported by the anthropic API
checking for context length limits not supported; anthropic have not released a tokenizer yet (like tiktoken from openai)
supports chat models only. if anthropic releases base models at some point, we will address that when it arises