search

julep-ai / julep

A new DSL and server for AI agents and multi-step tasks
https://julep.ai
Apache License 2.0
2.66k stars 911 forks source link

feat(agents-api): Improve error reporting for ValidationError on union types #801

Closed creatorrr closed 3 weeks ago

creatorrr commented 3 weeks ago

Previously, the error messages on validation errors were horrendous. The following input would have yielded a huge, unreadable mess as an error message. This happened because we make heavy use of union types and pydantic tries to unify each type and collects the error for each one of them. Fixed by returning only the deepest matching exceptions.

Input

name: Browserbase Task

tools:
- name: computer
  type: computer_20241022
  computer_20241022:
    display_height_px: 768
    display_width_px: 1024
    display_number: 1

- name: spider_crawler
  type: integration
  integration:
    provider: spider
    setup:
      spider_api_key: "sk-8d6285a0-7b9f-493d-bcf4-3e15e68b1304"

main:

- tool: spider_crawler
  arguments:
    url: "'https://julep.ai'"
- prompt:
    - role: useaar                    # <---- the problem
      content: hi

New error format

{'errors': ["Input should be 'user', 'assistant', 'system', 'function', 'function_response', 'function_call' or 'auto'"], 'offending_input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}, 'location': ['main', 1, 'PromptStep', 'prompt', 'list[PromptItem]', 0, 'role']}
{
    "errors": [
        "Input should be 'user', 'assistant', 'system', 'function', 'function_response', 'function_call' or 'auto'"
    ],
    "offending_input": {
        "prompt": [
            {
                "role": "useaar",
                "content": "hi"
            }
        ]
    },
    "location": [
        "main",
        1,
        "PromptStep",
        "prompt",
        "list[PromptItem]",
        0,
        "role"
    ]
}

Previous error format

[{'type': 'missing', 'loc': ('body', 'main', 1, 'EvaluateStep', 'evaluate'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'ToolCallStep', 'tool'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'literal_error', 'loc': ('body', 'main', 1, 'PromptStep', 'prompt', 'list[PromptItem]', 0, 'role'), 'msg': "Input should be 'user', 'assistant', 'system', 'function', 'function_response', 'function_call' or 'auto'", 'input': 'useaar', 'ctx': {'expected': "'user', 'assistant', 'system', 'function', 'function_response', 'function_call' or 'auto'"}}, {'type': 'string_type', 'loc': ('body', 'main', 1, 'PromptStep', 'prompt', 'str'), 'msg': 'Input should be a valid string', 'input': [{'role': 'useaar', 'content': 'hi'}]}, {'type': 'missing', 'loc': ('body', 'main', 1, 'GetStep', 'get'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'SetStep', 'set'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'LogStep', 'log'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'YieldStep', 'workflow'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'ReturnStep', 'return'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'SleepStep', 'sleep'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'ErrorWorkflowStep', 'error'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'WaitForInputStep', 'wait_for_input'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'IfElseWorkflowStep', 'if'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'IfElseWorkflowStep', 'then'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'SwitchStep', 'switch'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'ForeachStep', 'foreach'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'ParallelStep', 'parallel'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'Main', 'over'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}, {'type': 'missing', 'loc': ('body', 'main', 1, 'Main', 'map'), 'msg': 'Field required', 'input': {'prompt': [{'role': 'useaar', 'content': 'hi'}]}}]
[
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "EvaluateStep",
            "evaluate"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "ToolCallStep",
            "tool"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "literal_error",
        "loc": [
            "body",
            "main",
            1,
            "PromptStep",
            "prompt",
            "list[PromptItem]",
            0,
            "role"
        ],
        "msg": "Input should be 'user', 'assistant', 'system', 'function', 'function_response', 'function_call' or 'auto'",
        "input": "useaar",
        "ctx": {
            "expected": "'user', 'assistant', 'system', 'function', 'function_response', 'function_call' or 'auto'"
        }
    },
    {
        "type": "string_type",
        "loc": [
            "body",
            "main",
            1,
            "PromptStep",
            "prompt",
            "str"
        ],
        "msg": "Input should be a valid string",
        "input": [
            {
                "role": "useaar",
                "content": "hi"
            }
        ]
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "GetStep",
            "get"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "SetStep",
            "set"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "LogStep",
            "log"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "YieldStep",
            "workflow"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "ReturnStep",
            "return"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "SleepStep",
            "sleep"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "ErrorWorkflowStep",
            "error"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "WaitForInputStep",
            "wait_for_input"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "IfElseWorkflowStep",
            "if"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "IfElseWorkflowStep",
            "then"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "SwitchStep",
            "switch"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "ForeachStep",
            "foreach"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "ParallelStep",
            "parallel"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "Main",
            "over"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    },
    {
        "type": "missing",
        "loc": [
            "body",
            "main",
            1,
            "Main",
            "map"
        ],
        "msg": "Field required",
        "input": {
            "prompt": [
                {
                    "role": "useaar",
                    "content": "hi"
                }
            ]
        }
    }
]

[!IMPORTANT] Enhance error reporting for ValidationError and RequestValidationError in make_exception_handler by providing detailed error information.

  • Error Handling:
    • Enhance make_exception_handler in web.py to improve error reporting for ValidationError and RequestValidationError.
    • Extracts deepest matching errors, common location, and offending input for better clarity.
    • Returns only the error messages in the response.

This description was created by Ellipsis for 5292f57215413df342c92218a4a5c3f1d08d8a6e. It will automatically update as commits are pushed.

sweep-ai[bot] commented 3 weeks ago

Hey @creatorrr, here is an example of how you can ask me to improve this pull request:

@sweep Add unit tests for the `make_exception_handler` function that cover:
1. Validation errors with different depths of nested fields
2. Common location extraction from multiple errors
3. Offending input extraction for different data structures (dict, list)
4. Edge cases where location paths don't fully exist in the input

:book: For more information on how to use Sweep, please read our documentation.

Vedantsahai18 commented 3 weeks ago 
FAILED: /Users/Dell/Desktop/Julep_Workspace/julep/agents-api/.pytype/pyi/agents_api/web.pyi 
/Users/Dell/miniconda3/envs/julep/bin/python -m pytype.main --disable pyi-error --imports_info /Users/Dell/Desktop/Julep_Workspace/julep/agents-api/.pytype/imports/agents_api.web.imports --module-name agents_api.web --platform linux -V 3.12 -o /Users/Dell/Desktop/Julep_Workspace/julep/agents-api/.pytype/pyi/agents_api/web.pyi --analyze-annotated --nofail --none-is-not-bool --quick --strict-none-binding /Users/Dell/Desktop/Julep_Workspace/julep/agents-api/agents_api/web.py
/Users/Dell/Desktop/Julep_Workspace/julep/agents-api/agents_api/web.py:68:28: error: in _handler: unsupported operand type(s) for item retrieval: 'error: Exception' and ''loc': str' [unsupported-operands]
  No attribute '__getitem__' on 'error: Exception'

            max_depth = max(len(error["loc"]) for error in errors)
                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

/Users/Dell/Desktop/Julep_Workspace/julep/agents-api/agents_api/web.py:69:56: error: in _handler: unsupported operand type(s) for item retrieval: 'error: Exception' and ''loc': str' [unsupported-operands]
  No attribute '__getitem__' on 'error: Exception'

            errors = [error for error in errors if len(error["loc"]) == max_depth]
                                                       ~~~~~~~~~~~~

/Users/Dell/Desktop/Julep_Workspace/julep/agents-api/agents_api/web.py:72:24: error: in _handler: unsupported operand type(s) for item retrieval: 'Exception' and ''loc': str' [unsupported-operands]
  No attribute '__getitem__' on 'Exception'

            location = errors[0]["loc"]
                       ~~~~~~~~~~~~~~~~

/Users/Dell/Desktop/Julep_Workspace/julep/agents-api/agents_api/web.py:74:43: error: in _handler: unsupported operand type(s) for item retrieval: 'error: Exception' and ''loc': str' [unsupported-operands]
  No attribute '__getitem__' on 'error: Exception'

                for a, b in zip(location, error["loc"]):
                                          ~~~~~~~~~~~~

/Users/Dell/Desktop/Julep_Workspace/julep/agents-api/agents_api/web.py:83:31: error: in _handler: No attribute 'body' on Exception [attribute-error]

            offending_input = exc.body
                              ~~~~~~~~

/Users/Dell/Desktop/Julep_Workspace/julep/agents-api/agents_api/web.py:100:23: error: in _handler: unsupported operand type(s) for item retrieval: 'error: Exception' and ''msg': str' [unsupported-operands]
  No attribute '__getitem__' on 'error: Exception'

            errors = [error["msg"] for error in errors]
                      ~~~~~~~~~~~~

For more details, see https://google.github.io/pytype/errors.html
ninja: build stopped: cannot make progress due to previous errors.
Leaving directory '.pytype'
Error: Sequence aborted after failed subtask 'typecheck'