Haiku and Sonnet return incorrect types when calling tools with lists (Opus works)

[willbakst]

The following code snippet:

import anthropic

client = anthropic.Anthropic()

response = client.beta.tools.messages.create(
    model="claude-3-haiku-20240307",
    max_tokens=1024,
    tools=[
        {
            "name": "Books",
            "description": "Correctly formatted and typed parameters extracted from the completion. Must include required parameters and may exclude optional parameters unless present in the text.",
            "input_schema": {
                "type": "object",
                "properties": {
                    "book_list": {
                        "type": "array",
                        "items": {"type": "string"},
                    },
                },
                "required": ["book_list"],
            },
        }
    ],
    messages=[
        {
            "role": "user",
            "content": "Recommend me 3 self help books.",
        }
    ],
)

print(response)

Results in the following output:

ToolsBetaMessage(id='msg_01HdJb4aB14KbSaQdUa94gUj', content=[TextBlock(text='Here are 3 recommended self-help books:', type='text'), ToolUseBlock(id='toolu_01X2m1GBeiSz5VkLrYuKE41o', input={'book_list': '["The Power of Habit" by Charles Duhigg, "Atomic Habits" by James Clear, "Man\'s Search for Meaning" by Viktor Frankl]'}, name='Books', type='tool_use')], model='claude-3-haiku-20240307', role='assistant', stop_reason='tool_use', stop_sequence=None, type='message', usage=Usage(input_tokens=366, output_tokens=103))

Notice that book_list is a string and not a list. I tried this with Sonnet and got the same thing. Opus properly outputted a list as expected.

[willbakst]

Update: not ideal, but I can get correct behavior if I add "MUST BE A LIST" to my prompt. This is not tenable for more complex schemas or things that are more dynamic like agents.

[aaron-lerner]

Hey @willbakst, thanks for sharing your experience. We're aware of the issue and are working to improve how our models handle these types of inputs. Sorry there's no immediate solution here, but more to come!

[willbakst]

No worries! I appreciate the response and am always looking forward to what you've got cooking :)