UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://inspect.ai-safety-institute.org.uk/
MIT License
604 stars 112 forks source link

[Bug] Incorrect schema for vertex AI #402

Closed self-reasoning-evals closed 1 month ago

self-reasoning-evals commented 1 month ago

When trying to use a vertex AI model, I run into the following error

│ │***/lib/python3.11/site-packages/verte… │                                                                                 │
│ │ in __init__                                                                  │                                                                                 │
│ │                                                                              │                                                                                 │
│ │   1654 │   │   │   │   Model uses it to decide how and whether to call the f │                                                                                 │
│ │   1655 │   │   """                                                           │                                                                                 │
│ │   1656 │   │   gapic_schema_dict = _convert_schema_dict_to_gapic(parameters) │                                                                                 │
│ │ ❱ 1657 │   │   raw_schema = aiplatform_types.Schema(gapic_schema_dict)       │                                                                                 │
│ │   1658 │   │   self._raw_function_declaration = gapic_tool_types.FunctionDec │                                                                                 │
│ │   1659 │   │   │   name=name, description=description, parameters=raw_schema │                                                                                 │
│ │   1660 │   │   )                                                             │                                                                                 │
│ │                                                                              │                                                                                 │
│ │ ***/lib/python3.11/site-packages/proto… │                                                                                 │
│ │ in __init__                                                                  │                                                                                 │
│ │                                                                              │                                                                                 │
│ │   721 │   │   │   │   if ignore_unknown_fields:                              │                                                                                 │
│ │   722 │   │   │   │   │   continue                                           │                                                                                 │
│ │   723 │   │   │   │                                                          │                                                                                 │
│ │ ❱ 724 │   │   │   │   raise ValueError(                                      │                                                                                 │
│ │   725 │   │   │   │   │   "Unknown field for {}: {}".format(self.__class__._ │                                                                                 │
│ │   726 │   │   │   │   )                                                      │                                                                                 │
│ │   727                                                                        │                                                                                 │
│ ╰──────────────────────────────────────────────────────────────────────────────╯                                                                                 │
│ ValueError: Unknown field for Schema: additionalProperties                                                                                                       │
│                                                                                                                                                                  │
│ Task interrupted (no samples completed before interruption)               

It seems to be cause by the param additionalProperties which got added to ToolParams.

The easiest fix is probably to just remove additionalProperties here in vertex.py:

def chat_tools(tools: list[ToolInfo]) -> list[Tool]:
    declarations = [
        FunctionDeclaration(
            name=tool.name,
            description=tool.description,
            parameters=tool.parameters.model_dump(exclude_none=True),  # replace with .model_dump(exclude_none=True, exclude={'additionalProperties'})
        )
        for tool in tools
    ]
    return [Tool(declarations)]
jjallaire commented 1 month ago

Thanks for reporting this! Fixed here https://github.com/UKGovernmentBEIS/inspect_ai/pull/404