Converse API skips 'required' tool spec parameters

austinmw commented 3 months ago

Describe the bug

I've defined the following nested tool spec:

import boto3

formatted_tools = [
    {
        "toolSpec": {
            "name": "WorkoutPlan",
            "description": "Model for WorkoutPlan",
            "inputSchema": {
                "json": {
                    "type": "object",
                    "properties": {
                        "weeks": {
                            "type": "array",
                            "description": "List of 2 workout weeks.",
                            "items": {
                                "type": "object",
                                "properties": {
                                    "days": {
                                        "type": "array",
                                        "description": "List of 7 workout days.",
                                        "items": {
                                            "type": "object",
                                            "properties": {
                                                "description": {
                                                    "type": "string",
                                                    "description": "Description of the workout for this day."
                                                },
                                                "duration": {
                                                    "type": "integer",
                                                    "description": "Duration of the workout in minutes."
                                                }
                                            },
                                            "required": [
                                                "description",
                                                "duration"
                                            ]
                                        }
                                    }
                                },
                                "required": [
                                    "days"
                                ]
                            }
                        }
                    },
                    "required": [
                        "weeks"
                    ]
                }
            }
        }
    }
]

# Define input message
input_text = """
Create a workout program for the below person:

Age: 70
Weight: 240
Gender: Male
Height: 6'2"

I want the program to focus on cardio and weight loss.
"""
messages = [{"role": "user", "content": [{"text": input_text}]}]

bedrock_client = boto3.client(service_name='bedrock-runtime')

# Invoke converse API with tool config
response = bedrock_client.converse(
    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
    messages=messages,
    inferenceConfig={"maxTokens": 4096, "temperature": 0},
    toolConfig={"tools": formatted_tools},
)

assert response['stopReason'] == 'tool_use'

for content in response['output']['message']['content']:
    if 'toolUse' in content:
        tool = content['toolUse']
        print(f"Requesting tool {tool['name']}. Request: {tool['toolUseId']}")
        print(json.dumps(tool['input'], indent=4))  # This output often misses several 'required' fields (e.g. 'duration')

However the output often (depending on variability in the run) skips some required parameters. For example this output is missing several duration fields:

Requesting tool WorkoutPlan. Request: tooluse_aHTGParNQOiW13Fgf41XvQ
{
    "properties": {
        "weeks": [
            {
                "days": [
                    {
                        "description": "30 minutes low-impact cardio (walking or cycling)",
                        "duration": 30
                    },
                    {
                        "description": "Rest day"
                    },
                    {
                        "description": "30 minutes water aerobics or swimming laps",
                        "duration": 30
                    },
                    {
                        "description": "Light strength training - 2 sets of 10-12 reps of seated exercises targeting major muscle groups",
                        "duration": 45
                    },
                    {
                        "description": "45 minutes low-impact cardio (walking or stationary cycling)",
                        "duration": 45
                    },
                    {
                        "description": "Rest day"
                    },
                    {
                        "description": "Flexibility and balance training (stretching, yoga, tai chi)",
                        "duration": 30
                    }
                ]
            },
            {
                "days": [
                    {
                        "description": "30 minutes low-impact cardio (walking or recumbent cycling)",
                        "duration": 30
                    },
                    {
                        "description": "Rest day"
                    },
                    {
                        "description": "45 minutes water aerobics or lap swimming",
                        "duration": 45
                    },
                    {
                        "description": "Light strength training - 2 sets of 10-12 reps of seated/machine exercises",
                        "duration": 45
                    },
                    {
                        "description": "60 minutes low-impact cardio (walking or stationary cycling)",
                        "duration": 60
                    },
                    {
                        "description": "Rest day"
                    },
                    {
                        "description": "Flexibility and balance training (yoga, stretching)",
                        "duration": 30
                    }
                ]
            }
        ]
    }
}

Is there anything wrong with the toolSpec I've provided, is this a bug, or does the model/API just not support nested parameters?

Expected Behavior

Expecting every required parameter to be included

Current Behavior

Some required parameters are skipped

Reproduction Steps

See code

Possible Solution

Add documentation around how this works. I think many people might (wrongly?) assume that when using a toolSpec, outputs will be verified to ensure they exactly match a provided spec. Having missing parameters in the output is very surprising.

Note that I can get this to output all parameters more reliably by improving the descriptions. I'm more concerned about the lack of clarity or documentation here. One would expect indeterministic results for the values of the output that are filled in, but not in the keys as well!

Additional Information/Context

For context, I'm trying to define a toolSpec to match this Pydantic WorkoutPlan model:

# Define the Pydantic models
class WorkoutDay(BaseModel):
    description: str = Field(..., description="Description of the workout for this day.")
    duration: int = Field(..., description="Duration of the workout in minutes.")

class WorkoutWeek(BaseModel):
    days: List[WorkoutDay] = Field(..., description="List of 7 workout days.")

class WorkoutPlan(BaseModel):
    weeks: List[WorkoutWeek] = Field(..., description="List of 2 workout weeks.")

I have code that takes as input tools = [WorkoutPlan] and outputs the formatted_tools list I provided.

SDK version used

Latest

Environment details (OS name and version, etc.)

MacOS

tim-finnigan commented 3 months ago

Thanks for reaching out. The converse command makes a call to the underlying Bedrock Converse API, so this issue involves the API rather than Boto3 directly. This may just be a limitation with the Bedrock/Claude results being non-deterministic. I recommend reaching out to the Bedrock team internally for further information on this, along with the request to update the documentation and/or add a Converse example here: https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html#conversation-inference-examples. We could also track the general request to add more Bedrock examples here: https://github.com/boto/boto3/discussions/4222

github-actions[bot] commented 3 months ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

boto / boto3