[Upstream Bug]: Vertex AI - Gemini 1.5 Pro - 500 Internal error when using tool/function calling

[Manouchehri]

What happened?

Initially I thought this was a LiteLLM error, but doing the same request with curl fails too. Making this ticket so others don't waste their time. =)

curl -v -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  https://us-east4-aiplatform.googleapis.com/v1/projects/litellm-epic/locations/us-east4/publishers/google/models/gemini-1.5-pro-preview-0409:streamGenerateContent \
  -d '{
    "contents": [{
      "role": "user",
      "parts": [{
        "text": "Say the word hello 1 times."
      }]
    }],
    "tools": [{
      "function_declarations": [
        {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA or a zip code e.g. 95616"
              }
            },
            "required": [
              "location"
            ]
          }
        }
      ]
    }]
  }'

Relevant log output

< HTTP/2 500
< vary: X-Origin
< vary: Referer
< vary: Origin,Accept-Encoding
< content-type: application/json; charset=UTF-8
< date: Tue, 23 Apr 2024 16:06:59 GMT
< server: scaffolding on HTTPServer2
< cache-control: private
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< accept-ranges: none
<
[{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}
* Connection #0 to host us-east4-aiplatform.googleapis.com left intact

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

[ishaan-jaff]

is it specific to this exact request ?

[Manouchehri]

Sorta. A slightly different request like this works fine.

curl -v -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  https://us-east4-aiplatform.googleapis.com/v1/projects/litellm-epic/locations/us-east4/publishers/google/models/gemini-1.5-pro-preview-0409:streamGenerateContent \
  -d '{
    "contents": [{
      "role": "user",
      "parts": [{
        "text": "What time is it?"
      }]
    }],
    "tools": [{
      "function_declarations": [
        {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA or a zip code e.g. 95616"
              }
            },
            "required": [
              "location"
            ]
          }
        }
      ]
    }]
  }'

< HTTP/2 200
< content-type: application/json; charset=UTF-8
< vary: X-Origin
< vary: Referer
< vary: Origin,Accept-Encoding
< date: Tue, 23 Apr 2024 16:11:47 GMT
< server: scaffolding on HTTPServer2
< cache-control: private
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< accept-ranges: none
<

[{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "I"
          }
        ]
      }
    }
  ]
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": " am sorry, I cannot fulfill this request. The available tools lack the desired functionality"
          }
        ]
      },
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.05079946,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.098946586
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.0063883206,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.0045384318
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.045183755,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.01812472
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.041462272,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.0705596
        }
      ]
    }
  ]
}
,
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "."
          }
        ]
      },
      "finishReason": "STOP",
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.05089372,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.086632065
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.006121281,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.0040701376
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.046464667,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.016340127
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.043204036,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.07068779
        }
      ]
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 47,
    "candidatesTokenCount": 18,
    "totalTokenCount": 65
  }
}
]

[Manouchehri]

This fails (which is the default):

"tool_config": {
    "function_calling_config": {
        "mode":"AUTO"
    }
}

This works:

"tool_config": {
    "function_calling_config": {
        "mode":"ANY"
    }
}

And so does this:

"tool_config": {
    "function_calling_config": {
        "mode":"NONE"
    }
}

Given that AUTO should be a superset of ANY (forcing a function) and NONE (not using any functions), it definitely looks an upstream problem to me.