LiteLLM incompatibility

[JohanBekker]

• [x ] This is actually a bug report.
• [ ] I am not getting good LLM Results
• [ ] I have tried asking for help in the community on discord or discussions and have not received a response.
• [ ] I have tried searching the documentation and have not found an answer.

What Model are you using?

• [ ] gpt-3.5-turbo
• [ ] gpt-4-turbo
• [ ] gpt-4
• [x ] Other (please specify)

Environment Instructor 1.4.0 LiteLLM 1.44.4

Describe the bug While trying to extend our LLM applications with LiteLLM on top of Instructor for AWS Bedrock Anthropic models, I've come across 2 problems

• While using Mode.ANTHROPIC_JSON, an assertion error is raised as LiteLLM returns a ModelResponse type (https://github.com/BerriAI/litellm/blob/main/litellm/types/utils.py#L538), which in the instructor library is checked with an assertion to be of kind Message (https://github.com/jxnl/instructor/blob/main/instructor/function_calls.py#L205)
• While using Mode.MD_JSON, together with a predefined system message, this line https://github.com/jxnl/instructor/blob/main/instructor/process_response.py#L312 tries to extend the predefined system message assuming it is a string. However, LiteLLM seems to force the content of each message into a list, which causes that line to extend the list with a list entry per character in the Mode.MD_JSON system message, which leads to an error further down the line in LiteLLM This only appears to happens if one of the messages has a list as content, but as I understand it, the list is necessary when sending an image to a vLLM.

{'role': 'system', 'content': [{'type': 'text', 'text': 'You are a financial analyst. You have been given a chart to analyze. Extract the data from this chart.'}, '\n', '\n', '\n', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'A', 's', ' ', 'a', ' ', 'g', 'e', 'n', 'i', 'u', 's', ' ', 'e', 'x', 'p', 'e', 'r', 't', ',', ' ', 'y', 'o', 'u', 'r', ' ', 't', 'a', 's', 'k', ' ', 'i', 's', ' ', 't', 'o', ' ', 'u', 'n', 'd', 'e', 'r', 's', 't', 'a', 'n', 'd',  ... etc

To Reproduce

You can take a random image to reproduce. Both issues can be reproduced with this snippet, just switch out the Mode.

import asyncio
import base64
import os
from typing import Dict

import instructor
from dotenv import load_dotenv
from instructor import AsyncInstructor
from litellm import Router
from pydantic import BaseModel, Field

load_dotenv()

class ChartData(BaseModel):
    """Extracted data from a financial chart"""

    chart_type: str = Field("", description="The type of chart")
    data: Dict[str, list[float]] = Field({}, description="The extracted data")

def get_instructor():
    router = Router(
        model_list=[
            {
                "model_name": "claude-sonnet-3.5",
                "litellm_params": {
                    "model": "bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0",
                    "aws_access_key_id": os.getenv("AWS_ACCESS_KEY_ID"),
                    "aws_secret_access_key": os.getenv("AWS_SECRET_ACCESS_KEY"),
                    "aws_region_name": os.getenv("AWS_DEFAULT_REGION"),
                },
            },
            # Add more models if needed
        ],
    )
    return instructor.from_litellm(
        router.acompletion,
        # mode=instructor.Mode.MD_JSON,  # For issue #2
        mode=instructor.Mode.ANTHROPIC_JSON,  # For issue #1
    )

async def call_llm(llm: AsyncInstructor, encoded_image: str):
    response = await llm.chat.completions.create(
        model="claude-sonnet-3.5",
        response_model=ChartData,
        messages=[
            {
                "role": "system",
                "content": "You are a financial analyst. You have been given a chart to analyze. Extract the data from this chart.",
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Extract data from this image."},
                    {
                        "type": "image_url",
                        "image_url": {"url": "data:image/jpeg;base64," + encoded_image},
                    },
                ],
            },
        ],
    )
    return response

def load_and_encode_image(image_path: str) -> str:
    with open(image_path, "rb") as image_file:
        encoded_image = base64.b64encode(image_file.read()).decode("utf-8")
    return encoded_image

async def extract_chart_data():
    image = load_and_encode_image("testchart.jpg")
    instructor = get_instructor()
    chart_data = await call_llm(instructor, image)

    return chart_data

if __name__ == "__main__":
    response = asyncio.run(extract_chart_data())

Expected behavior I expect LiteLLM to work with Instructor for Bedrock Anthropic models, for both LLM and vLLM use cases.

Error log Issue 1:

Traceback (most recent call last):
  File "/home/jbekker/miniconda3/envs/zenith/lib/python3.11/site-packages/instructor/retry.py", line 242, in retry_async
    return await process_response_async(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jbekker/miniconda3/envs/zenith/lib/python3.11/site-packages/instructor/process_response.py", line 77, in process_response_async
    model = response_model.from_response(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jbekker/miniconda3/envs/zenith/lib/python3.11/site-packages/instructor/function_calls.py", line 124, in from_response
    return cls.parse_anthropic_json(completion, validation_context, strict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jbekker/miniconda3/envs/zenith/lib/python3.11/site-packages/instructor/function_calls.py", line 205, in parse_anthropic_json
    assert isinstance(completion, Message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Issue 2:

litellm.exceptions.APIConnectionError: litellm.APIConnectionError: 'str' object has no attribute 'get'
Traceback (most recent call last):
  File "/home/jbekker/miniconda3/envs/zenith/lib/python3.11/site-packages/litellm/main.py", line 2303, in completion
    response = bedrock_converse_chat_completion.completion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jbekker/miniconda3/envs/zenith/lib/python3.11/site-packages/litellm/llms/bedrock_httpx.py", line 1586, in completion
    if m.get("type", "") == "text" and len(m["text"]) > 0:
       ^^^^^
AttributeError: 'str' object has no attribute 'get'

Happy to contribute if necessary.

[jxnl]

open to take a PR that replaces the isinstance check with some hasattr checks