search

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://inspect.ai-safety-institute.org.uk/
MIT License
612 stars 113 forks source link

Feature Request: Support Anthropic on Vertex models. #574

Open simonbutt opened 1 month ago

simonbutt commented 1 month ago

Currently Anthropic models on AWS Bedrock is supported but not Google Vertex.

"As a user of Inspect eval framework and also GCP, I'd like to use Anthropic models through Vertex AI in the Google Cloud project and region of my choice, rather than having to create a new billing process to use through the Anthropic API"

Details on all anthropic models: https://docs.anthropic.com/en/docs/about-claude/models

jjallaire-aisi commented 1 month ago

I don't have access to Vertex to test, however I just made a change that should enable you to put together a PR for this. The Anthropic client already supports Bedrock, and I just made it possible to use an anthropic/bedrock/ model name prefix to opt into using this client: https://github.com/UKGovernmentBEIS/inspect_ai/pull/593/commits/ca04ef5ef5c0fe6497fa1b9d90e1d7a174123377

Anthropic also has a built-in Vertex client, so you could add a vertex flag (and service name check) that binds to Vertex rather than Bedrock or the default Anthropic endpoint.