This PR contributes a TogetherSolver class, a solver for using models served by the Together AI API
Because Together supports the OpenAI python sdk, we simply create a subclass of the OpenAISolver, overriding some functionality. There is therefore some refactoring of the OpenAISolver included in this PR to facilitate this code sharing.
At the moment, we support the models specified in evals/registry/solvers/together.yaml, but in principle most models offered from the Together AI API can easily be added
Notes:
logit biasing not supported by the Together API due to a lack of a unified tokenizer a la tiktoken from openai
For the same reason, checking for context length limits not supported
This PR contributes a
TogetherSolverclass, a solver for using models served by the Together AI APIBecause Together supports the OpenAI python sdk, we simply create a subclass of the
OpenAISolver, overriding some functionality. There is therefore some refactoring of theOpenAISolverincluded in this PR to facilitate this code sharing.At the moment, we support the models specified in
evals/registry/solvers/together.yaml, but in principle most models offered from the Together AI API can easily be addedNotes: