[Feature]: DAY 1 SUPPORT Add Claude -3

[ishaan-jaff]

The Feature

Motivation, pitch

New model

Twitter / LinkedIn details

No response

[ishaan-jaff]

We'll need to add the messages endpoint for claude3 - it fails on their completion endpoint

Error calling model: AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}

[krrishdholakia]

No bedrock support yet.

[krrishdholakia]

I don't think the legacy text completion endpoint supports claude 3. We should move to using the anthropic 2.1 endpoint - https://github.com/BerriAI/litellm/issues/1209

[krrishdholakia]

You can also send images in claude 3 - will need to add pricing for this

[krrishdholakia]

support tool calling - https://github.com/anthropics/anthropic-cookbook/blob/main/function_calling/function_calling.ipynb

[ishaan-jaff]

Action Items:

• Migrate to /messages + add claude-3 @ishaan-jaff
• Add tool calling / images @krrishdholakia

[ishaan-jaff]

PR to add initial support : https://github.com/BerriAI/litellm/pull/2315

[ishaan-jaff]

Model Pricing info here for reference https://docs.anthropic.com/claude/docs/models-overview#meet-claude-3

[krrishdholakia]

I'll take care of the image + tool calling cost tracking in my PR

[krrishdholakia]

[ishaan-jaff]

Notes about the /messages endpoint

• Anthropic Deprecated support for claude-instant-1 on the new messages endpoint

1. Anthropic supports roles like "user" and "assistant",
2. The first message always needs to be of role "user"
3. Each message must alternate between "user" and "assistant"

3. Each message must alternate between "user" and "assistant"

For v0 PR not addressing no.3

[ashot]

Sorry if this is a silly question but does day 1 support mean you guy are hoping to get this done today? Just wondering if I should attempt to hack something myself.
This project is awesome btw.

[ishaan-jaff]

@ashot it will be live in 1-2 hours, PR for v0 support is here: https://github.com/BerriAI/litellm/pull/2315/files

[ashot]

love it, thank you!

[haseeb-heaven]

Wow great one waiting to test Claude-3 in my code interpreter.

[ishaan-jaff]

Working v0 PR is live here: https://github.com/BerriAI/litellm/commit/14fc8355fb1ce89a143ebb7b3171dfa19cb46d4e

[krrishdholakia]

@ishaan-jaff is this ready for me to stack the tool calling + image calling PRs on this?

[ishaan-jaff]

Yes @krrishdholakia

[krrishdholakia]

@ishaan-jaff looks like there's a separate param for system prompt

[krrishdholakia]

V1 needs to add:

• [ ] support for system prompt
• [ ] Support for tool calling
• [ ] Support for image calls

[ishaan-jaff]

confirmed 1.28.13 has v0 support working

[ishaan-jaff]

@ishaan-jaff looks like there's a separate param for system prompt

oh missed this, will your v1 PR cover this @krrishdholakia ? else I can make the fix

[krrishdholakia]

yea i'm fixing it for tool calling

[ishaan-jaff]

FYI Anthropic deprecated the following models on /messages

• claude-2 (anthropic recommends using claude-2.1)
• claude-instant-1 (anthropic recommends using claude-instant-1.2)

Since we moved everything to /messages this means litellm will not support claude-2 and claude-instant-1

[krrishdholakia]

That's funny - claude instant 1.2 works

[ishaan-jaff]

Yup - Lets put a notice on the latest release about this and see if users want us to maintain - Claude-instant-1 and Claude-2

[krrishdholakia]

should we just have backwards compatibility for those 2 models? would be a bit weird to suddenly start failing calls

[krrishdholakia]

since we have the code, it's just routing no?

[krrishdholakia]

if claude-instant-1 or claude-2 route to old_anthropic.py else route to anthropic.py

[krrishdholakia]

i always find it a bit stressful when together ai does this - and randomly kills access to models

[ishaan-jaff]

would be a bit weird to suddenly start failing calls

agree - bedrock did this to me and I didn't like it

[ishaan-jaff]

Action item 3: Maintain claude-instant-1 or claude-2 @ishaan-jaff

[krrishdholakia]

cool - let's just have a separate file called old anthropic - can you do this? @ishaan-jaff

[krrishdholakia]

thanks

[themrzmaster]

does it work with bedrock?

[krrishdholakia]

no - bedrock is not yet supported by anthropic

[krrishdholakia]

system prompt + tool calling support added - https://github.com/BerriAI/litellm/pull/2320

Now adding vision support

[themrzmaster]

what? it is

[ishaan-jaff]

@themrzmaster can you share the code snippet to call claude 3 on the AWS console? We don't have access on our AWS account

[themrzmaster]

{
  "modelId": "anthropic.claude-3-sonnet-20240229-v1:0",
  "contentType": "application/json",
  "accept": "application/json",
  "body": {
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 1000,
    "messages": {
      "role": "user",
      "content": [
        {
          "type": "image",
          "source": {
            "type": "base64",
            "media_type": "image/jpeg",
            "data": "iVBORw..."
          }
        },
        {
          "type": "text",
          "text": "What's in this image?"
        }
      ]
    }
  }
}

[krrishdholakia]

You're right @themrzmaster seems like claude and bedrock have conflicting info

[krrishdholakia]

Will work on adding bedrock support too @themrzmaster

might need some help testing it as i'm still waiting on access

can you help?

[themrzmaster]

@krrishdholakia yep! thanks

[krrishdholakia]

Let me finish up the anthropic integration, and i'll cc you on the bedrock pr.

cc: @dleen if you want to test out the bedrock/claude 3 integration as well

[krrishdholakia]

Got access to claude 3 - i should be able to test this on my end too

[ishaan-jaff]

🎢

[ishaan-jaff]

PR to maintain support for claude-instant-1 and claude-2 is here: https://github.com/BerriAI/litellm/pull/2321

[haseeb-heaven]

Are you trying to add all these Claude-3 Models Opus and Sonnet

[xihajun]

@krrishdholakia may I kindly ask how can we install the draft version and have a test? When I do poetry install on that repo, it doesn't help

[ishaan-jaff]

It's on here @xihajun https://github.com/BerriAI/litellm/releases/tag/v1.28.13