[Bug]: claude-3-5-sonnet-20240620 tool call streaming arguments has extra "{}" in front

[lawrencecchen]

What happened?

Note the arguments value -- there's an extra "{}" at the start.

Relevant log output

https://gist.github.com/lawrencecchen/e6b8fd8f06800006441a581b68d624c2

Twitter / LinkedIn details

@lawrencecchen

[krrishdholakia]

Looks like anthropic returns an empty {} at the start of a content_block

Let me check how openai handles this

[krrishdholakia]

Looks like openai just returns an empty string

{
    "id": "chatcmpl-9hQ3o34PxdNpDjnX1L8uitt3H2JSg",
    "object": "chat.completion.chunk",
    "created": 1720135308,
    "model": "gpt-4-turbo-2024-04-09",
    "system_fingerprint": "fp_486730399b",
    "choices": [
        {
            "index": 0,
            "delta": {
                "role": "assistant",
                "content": null,
                "tool_calls": [
                    {
                        "index": 0,
                        "id": "call_TB0wOPDkvPz2d8VGGaHW99JI",
                        "type": "function",
                        "function": {
                            "name": "get_current_weather",
                            "arguments": ""
                        }
                    }
                ]
            },
            "logprobs": null,
            "finish_reason": null
        }
    ]
}

[krrishdholakia]

@lawrencecchen Fixed - https://github.com/BerriAI/litellm/commit/f2dabc65be3db4a861e968986642eca7275e73e9

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"Al","role":"assistant"}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"right, let's"}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":" check"}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":" the weather in Boston"}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":" today:"}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"id":"toolu_019mYkJiJLhqRXvjTczBoh6b","function":{"arguments":"","name":"get_current_weather"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":""},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"{\"locat"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"ion\":"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":" \"Bost"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"on,"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":" MA\""},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":", \"unit\":"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":" \""},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"fahr"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"enh"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"eit"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"\"}"},"type":"function","index":0}]}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-8a5de1ec-2179-46f4-b00f-59d5edc5ad91","choices":[{"finish_reason":"tool_calls","index":0,"delta":{}}],"created":1720135382,"model":"claude-3-haiku-20240307","object":"chat.completion.chunk"}

data: [DONE]

[krrishdholakia]

Curious, are you trying to use the proxy in production? @lawrencecchen