search

sblakey / llm-bedrock-anthropic

Plugin for https://llm.datasette.io/en/stable/ to enable talking with Claude Instant and ClaudeV2 models on AWS Bedrock
Apache License 2.0
40 stars 12 forks source link

Claude 3.5 Sonnet modelId issue #20

Open jimmybutton opened 3 days ago

jimmybutton commented 3 days ago

The new Claude 3.5 Sonnet v2 doens't work for me:

llm -m bedrock-claude-sonnet-v2 "write a haiku about the clever octopus."
# Error: An error occurred (ValidationException) when calling the ConverseStream operation: 
# Invocation of model ID anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput 
# isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.

When I add a us. at the beginning (i.e. full model id "us.anthropic.claude-3-5-sonnet-20241022-v2:0"), then it works.

Anyone else having this issue? Maybe it's a regional thing? (I'm based in Europe)

jimmybutton commented 3 days ago

Just spotted this: My AWS console says that both 3.5 Sonnet v2 and 2.5 Haiku can only be used through an inference profile. So in order for me to access them, I need to have the us. at the beginning of the model id.

jimmybutton commented 3 days ago

The same issue was already mentioned by @alonw-cf here.