search

firebase / genkit

An open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations. Genkit works with various models and platforms.
Apache License 2.0
782 stars 118 forks source link

[JS] Schema validation failed when using a tool #889

Open marcelpinto opened 2 months ago

marcelpinto commented 2 months ago

Describe the bug I have a flow with a prompt defined with dotpromt with a json schema as output. When not using any tool the flow works fine, but when I added a tool the flow is failling because the Schema validation fails when the model replies that needs to use a tool.

To Reproduce

  1. Create a prompt with JSON schema
  2. Create a tool
  3. Make a request that forces the model to use the tool.

Expected behavior GenKit handles the tool calling, calls the tool and calls again the model with the result.

Actual behavior The flow fails with:

Candidate[0] Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:

- (root): must be object

Provided data:

null

Required JSON schema:
...

Runtime (please complete the following information): MacOS

** Node version v20.15.0

Additional context I tried to use the returnToolRequests: true, to handle the reply myself but the same issue happens since the check is on the model reply.

See trace logs: b198423822c2d17f302dbd4674c0b6cc.txt

MichaelDoyle commented 2 months ago

Thanks for reporting. Can you re-upload the trace as plain text? I am unable to read it.

marcelpinto commented 2 months ago

I saw that it had a weird format, but I just used the "export trace > save" option in the UI. Maybe another issue to report.

In any case, here is the trace.

I think the issue is in the generate.ts at the candidateError check: https://github.com/firebase/genkit/blob/2f6c7d8f3a9b7f0a9466d20598fa2cfb4363956d/js/ai/src/generate.ts#L668

if (c.text() === "" && c.data() === null) return null

c.data() is actually undefined. If I switch to:

if (c.text() === "" && !c.data()) return null

Then it works fine. 

{
  "traceId": "5569630ab920a351608eb119092ac5a1",
  "spans": {
    "e7cd70f774571360": {
      "spanId": "e7cd70f774571360",
      "traceId": "5569630ab920a351608eb119092ac5a1",
      "parentSpanId": "bfefc0be23fe76a3",
      "startTime": 1726056474355,
      "endTime": 1726056474367.9668,
      "attributes": {
        "genkit:type": "dotprompt",
        "genkit:name": "bot",
        "genkit:input": "{\"input\":{\"uid\":\"ipSL4LxtW325ME9KHXg02fvgGlGv\",\"text\":\"create a task for fixing a bug in the landing\"},\"tools\":[null],\"returnToolRequests\":true}",
        "genkit:path": "/{dev-run-action-wrapper}/{botFlow,t:action}/{botFlow,t:flow}/{bot,t:dotprompt}",
        "genkit:metadata:prompt_fingerprint": "dd8287bae646c972996f69604a9c7f6877b4ddf72a79f85924726e65874ef817",
        "genkit:output": "{\"model\":\"vertexai/gemini-1.5-pro\",\"config\":{\"temperature\":0.5},\"history\":[{\"role\":\"system\",\"content\":[{\"text\":\"\\nYou are a helpful AI task management assistant that takes the user input (text, voice or image) and\\nperforms actions on behalf of the user to organize and manage tasks in a project or answers questions\\nabout the projects tasks. Identify the type of message, perform the action and/or reply to it using the provided tools\\nIf possible provide 3 to 4 suggestion replies to the user input.\\n\\n\"}]}],\"prompt\":[{\"text\":\"\\nUseUser message: create a task for fixing a bug in the landing\\n\"}],\"candidates\":1,\"output\":{\"format\":\"json\",\"schema\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"},\"text\":{\"type\":\"string\",\"description\":\"The reply to the user message\"},\"transcript\":{\"type\":[\"string\",\"null\"],\"description\":\"The transcript of the user message, if the input was a voice message\"},\"suggestions\":{\"anyOf\":[{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"text\":{\"type\":\"string\",\"description\":\"The text of the suggestion to show to the user\"},\"type\":{\"type\":\"string\",\"enum\":[\"full\",\"partial\",\"action\"],\"description\":\"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"},\"action\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"}},\"required\":[\"text\",\"type\",\"action\"],\"additionalProperties\":true},\"description\":\"A set of flow up suggestions for the reply\"},{\"type\":\"null\"}],\"description\":\"A set of flow up suggestions for the reply\"},\"taskId\":{\"type\":[\"string\",\"null\"],\"description\":\"The task ID the reply type is associated with, or none if not applicable\"}},\"required\":[\"type\",\"text\"],\"additionalProperties\":true,\"$schema\":\"http://json-schema.org/draft-07/schema#\"},\"jsonSchema\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"},\"text\":{\"type\":\"string\",\"description\":\"The reply to the user message\"},\"transcript\":{\"type\":[\"string\",\"null\"],\"description\":\"The transcript of the user message, if the input was a voice message\"},\"suggestions\":{\"anyOf\":[{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"text\":{\"type\":\"string\",\"description\":\"The text of the suggestion to show to the user\"},\"type\":{\"type\":\"string\",\"enum\":[\"full\",\"partial\",\"action\"],\"description\":\"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"},\"action\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"}},\"required\":[\"text\",\"type\",\"action\"],\"additionalProperties\":true},\"description\":\"A set of flow up suggestions for the reply\"},{\"type\":\"null\"}],\"description\":\"A set of flow up suggestions for the reply\"},\"taskId\":{\"type\":[\"string\",\"null\"],\"description\":\"The task ID the reply type is associated with, or none if not applicable\"}},\"required\":[\"type\",\"text\"],\"additionalProperties\":true,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}},\"tools\":[null],\"returnToolRequests\":true}",
        "genkit:state": "success"
      },
      "displayName": "bot",
      "links": [],
      "instrumentationLibrary": {
        "name": "genkit-tracer",
        "version": "v1"
      },
      "spanKind": "INTERNAL",
      "sameProcessAsParentSpan": {
        "value": true
      },
      "status": {
        "code": 0
      },
      "timeEvents": {
        "timeEvent": []
      }
    },
    "973a66090b6e8dd0": {
      "spanId": "973a66090b6e8dd0",
      "traceId": "5569630ab920a351608eb119092ac5a1",
      "parentSpanId": "bfefc0be23fe76a3",
      "startTime": 1726056474400,
      "endTime": 1726056476317.246,
      "attributes": {
        "genkit:type": "action",
        "genkit:name": "vertexai/gemini-1.5-pro",
        "genkit:path": "/{dev-run-action-wrapper}/{botFlow,t:action}/{botFlow,t:flow}/{vertexai/gemini-1.5-pro,t:action,s:model}",
        "genkit:input": "{\"messages\":[{\"role\":\"system\",\"content\":[{\"text\":\"\\nYou are a helpful AI task management assistant that takes the user input (text, voice or image) and\\nperforms actions on behalf of the user to organize and manage tasks in a project or answers questions\\nabout the projects tasks. Identify the type of message, perform the action and/or reply to it using the provided tools\\nIf possible provide 3 to 4 suggestion replies to the user input.\\n\\n\"}]},{\"role\":\"user\",\"content\":[{\"text\":\"\\nUseUser message: create a task for fixing a bug in the landing\\n\"},{\"text\":\"\\n\\nOutput should be in JSON format and conform to the following schema:\\n\\n```\\n{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"type\\\":{\\\"type\\\":\\\"string\\\",\\\"enum\\\":[\\\"question\\\",\\\"update\\\",\\\"add\\\",\\\"remove\\\",\\\"breakdown\\\"],\\\"description\\\":\\\"The type of reply:\\\\n- question: the user asked a question and this is the reply.\\\\n - update: a task was updated.\\\\n - add: a task was added.\\\\n - remove: a task was removed.\\\\n - breakdown: a task was broken down.\\\"},\\\"text\\\":{\\\"type\\\":\\\"string\\\",\\\"description\\\":\\\"The reply to the user message\\\"},\\\"transcript\\\":{\\\"type\\\":[\\\"string\\\",\\\"null\\\"],\\\"description\\\":\\\"The transcript of the user message, if the input was a voice message\\\"},\\\"suggestions\\\":{\\\"anyOf\\\":[{\\\"type\\\":\\\"array\\\",\\\"items\\\":{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"text\\\":{\\\"type\\\":\\\"string\\\",\\\"description\\\":\\\"The text of the suggestion to show to the user\\\"},\\\"type\\\":{\\\"type\\\":\\\"string\\\",\\\"enum\\\":[\\\"full\\\",\\\"partial\\\",\\\"action\\\"],\\\"description\\\":\\\"The type of the suggestion:\\\\n- full: The full text to reply directly\\\\n- partial: The starting of the reply to be completed by the user\\\\n- action: An action to be taken\\\"},\\\"action\\\":{\\\"type\\\":\\\"string\\\",\\\"enum\\\":[\\\"question\\\",\\\"update\\\",\\\"add\\\",\\\"remove\\\",\\\"breakdown\\\"],\\\"description\\\":\\\"The type of reply:\\\\n- question: the user asked a question and this is the reply.\\\\n - update: a task was updated.\\\\n - add: a task was added.\\\\n - remove: a task was removed.\\\\n - breakdown: a task was broken down.\\\"}},\\\"required\\\":[\\\"text\\\",\\\"type\\\",\\\"action\\\"],\\\"additionalProperties\\\":true},\\\"description\\\":\\\"A set of flow up suggestions for the reply\\\"},{\\\"type\\\":\\\"null\\\"}],\\\"description\\\":\\\"A set of flow up suggestions for the reply\\\"},\\\"taskId\\\":{\\\"type\\\":[\\\"string\\\",\\\"null\\\"],\\\"description\\\":\\\"The task ID the reply type is associated with, or none if not applicable\\\"}},\\\"required\\\":[\\\"type\\\",\\\"text\\\"],\\\"additionalProperties\\\":true,\\\"$schema\\\":\\\"http://json-schema.org/draft-07/schema#\\\"}\\n```\\n\",\"metadata\":{\"purpose\":\"output\",\"source\":\"default\"}}]}],\"candidates\":1,\"config\":{\"temperature\":0.5},\"tools\":[{\"name\":\"manageTask\",\"description\":\"Create/Update/Remove a task. To create, provide a the task information (no ID), to update provide the ID and the task fields that needs to be updates, to remove provide only the Id. Returns the ID of the task.\",\"outputSchema\":{\"type\":\"string\",\"$schema\":\"http://json-schema.org/draft-07/schema#\"},\"inputSchema\":{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\",\"enum\":[\"create\",\"update\",\"remove\"]},\"id\":{\"type\":\"string\",\"description\":\"the ID of the task\"},\"task\":{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\",\"description\":\"the new title of the task\"},\"description\":{\"type\":\"string\",\"description\":\"the new description of the task\"},\"type\":{\"type\":\"string\",\"enum\":[\"bug\",\"feature\",\"idea\",\"task\"],\"description\":\"the type of task (if unclear use task)\"},\"status\":{\"type\":\"string\",\"description\":\"the new title of the task\"},\"transcript\":{\"type\":\"string\",\"description\":\"the new title of the task\"}},\"additionalProperties\":true,\"description\":\"A task details to be updated or created\"}},\"required\":[\"action\"],\"additionalProperties\":true,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}}],\"output\":{\"format\":\"json\",\"schema\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"},\"text\":{\"type\":\"string\",\"description\":\"The reply to the user message\"},\"transcript\":{\"type\":[\"string\",\"null\"],\"description\":\"The transcript of the user message, if the input was a voice message\"},\"suggestions\":{\"anyOf\":[{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"text\":{\"type\":\"string\",\"description\":\"The text of the suggestion to show to the user\"},\"type\":{\"type\":\"string\",\"enum\":[\"full\",\"partial\",\"action\"],\"description\":\"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"},\"action\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"}},\"required\":[\"text\",\"type\",\"action\"],\"additionalProperties\":true},\"description\":\"A set of flow up suggestions for the reply\"},{\"type\":\"null\"}],\"description\":\"A set of flow up suggestions for the reply\"},\"taskId\":{\"type\":[\"string\",\"null\"],\"description\":\"The task ID the reply type is associated with, or none if not applicable\"}},\"required\":[\"type\",\"text\"],\"additionalProperties\":true,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}}}",
        "genkit:metadata:subtype": "model",
        "genkit:output": "{\"candidates\":[{\"index\":0,\"message\":{\"role\":\"model\",\"content\":[{\"toolRequest\":{\"name\":\"manageTask\",\"input\":{\"task\":\"unknown\",\"action\":\"create\"}}}]},\"finishReason\":\"stop\",\"custom\":{\"safetyRatings\":[{\"category\":\"HARM_CATEGORY_HATE_SPEECH\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.22363281,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.10253906},{\"category\":\"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.47265625,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.16601563},{\"category\":\"HARM_CATEGORY_HARASSMENT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.29101563,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.12988281},{\"category\":\"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.1640625,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.1484375}]}}],\"custom\":{\"candidates\":[{\"content\":{\"role\":\"model\",\"parts\":[{\"functionCall\":{\"name\":\"manageTask\",\"args\":{\"task\":\"unknown\",\"action\":\"create\"}}}]},\"finishReason\":\"STOP\",\"safetyRatings\":[{\"category\":\"HARM_CATEGORY_HATE_SPEECH\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.22363281,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.10253906},{\"category\":\"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.47265625,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.16601563},{\"category\":\"HARM_CATEGORY_HARASSMENT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.29101563,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.12988281},{\"category\":\"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.1640625,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.1484375}],\"avgLogprobs\":-0.034768544137477875,\"index\":0}],\"usageMetadata\":{\"promptTokenCount\":591,\"candidatesTokenCount\":6,\"totalTokenCount\":597}},\"usage\":{\"inputCharacters\":2225,\"inputImages\":0,\"inputVideos\":0,\"inputAudioFiles\":0,\"outputCharacters\":0,\"outputImages\":0,\"outputVideos\":0,\"outputAudioFiles\":0,\"inputTokens\":591,\"outputTokens\":6,\"totalTokens\":597},\"latencyMs\":1913.7462920000107}",
        "genkit:state": "success"
      },
      "displayName": "vertexai/gemini-1.5-pro",
      "links": [],
      "instrumentationLibrary": {
        "name": "genkit-tracer",
        "version": "v1"
      },
      "spanKind": "INTERNAL",
      "sameProcessAsParentSpan": {
        "value": true
      },
      "status": {
        "code": 0
      },
      "timeEvents": {
        "timeEvent": []
      }
    },
    "bfefc0be23fe76a3": {
      "spanId": "bfefc0be23fe76a3",
      "traceId": "5569630ab920a351608eb119092ac5a1",
      "parentSpanId": "2d6b7e89cdf683d3",
      "startTime": 1726056474354,
      "endTime": 1726056476361.4963,
      "attributes": {
        "genkit:type": "flow",
        "genkit:name": "botFlow",
        "genkit:isRoot": true,
        "genkit:path": "/{dev-run-action-wrapper}/{botFlow,t:action}/{botFlow,t:flow}",
        "genkit:metadata:flow:execution": "0",
        "genkit:metadata:flow:name": "botFlow",
        "genkit:metadata:flow:id": "89b51b52-4543-4840-9729-fb7aaf72c64d",
        "genkit:metadata:flow:dispatchType": "start",
        "genkit:metadata:flow:state": "error",
        "genkit:input": "{\"uid\":\"ipSL4LxtW325ME9KHXg02fvgGlGv\",\"text\":\"create a task for fixing a bug in the landing\"}",
        "genkit:state": "error"
      },
      "displayName": "botFlow",
      "links": [],
      "instrumentationLibrary": {
        "name": "genkit-tracer",
        "version": "v1"
      },
      "spanKind": "INTERNAL",
      "sameProcessAsParentSpan": {
        "value": true
      },
      "status": {
        "code": 2,
        "message": "FAILED_PRECONDITION: Generation resulted in no candidates matching provided output schema.\n\nCandidate[0] Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- (root): must be object\n\nProvided data:\n\nnull\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"type\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"question\",\n        \"update\",\n        \"add\",\n        \"remove\",\n        \"breakdown\"\n      ],\n      \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n    },\n    \"text\": {\n      \"type\": \"string\",\n      \"description\": \"The reply to the user message\"\n    },\n    \"transcript\": {\n      \"type\": [\n        \"string\",\n        \"null\"\n      ],\n      \"description\": \"The transcript of the user message, if the input was a voice message\"\n    },\n    \"suggestions\": {\n      \"anyOf\": [\n        {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"text\": {\n                \"type\": \"string\",\n                \"description\": \"The text of the suggestion to show to the user\"\n              },\n              \"type\": {\n                \"type\": \"string\",\n                \"enum\": [\n                  \"full\",\n                  \"partial\",\n                  \"action\"\n                ],\n                \"description\": \"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"\n              },\n              \"action\": {\n                \"type\": \"string\",\n                \"enum\": [\n                  \"question\",\n                  \"update\",\n                  \"add\",\n                  \"remove\",\n                  \"breakdown\"\n                ],\n                \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n              }\n            },\n            \"required\": [\n              \"text\",\n              \"type\",\n              \"action\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"description\": \"A set of flow up suggestions for the reply\"\n        },\n        {\n          \"type\": \"null\"\n        }\n      ],\n      \"description\": \"A set of flow up suggestions for the reply\"\n    },\n    \"taskId\": {\n      \"type\": [\n        \"string\",\n        \"null\"\n      ],\n      \"description\": \"The task ID the reply type is associated with, or none if not applicable\"\n    }\n  },\n  \"required\": [\n    \"type\",\n    \"text\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}"
      },
      "timeEvents": {
        "timeEvent": [
          {
            "time": 1726056476360.4177,
            "annotation": {
              "attributes": {
                "exception.type": "Error",
                "exception.message": "FAILED_PRECONDITION: Generation resulted in no candidates matching provided output schema.\n\nCandidate[0] Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- (root): must be object\n\nProvided data:\n\nnull\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"type\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"question\",\n        \"update\",\n        \"add\",\n        \"remove\",\n        \"breakdown\"\n      ],\n      \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n    },\n    \"text\": {\n      \"type\": \"string\",\n      \"description\": \"The reply to the user message\"\n    },\n    \"transcript\": {\n      \"type\": [\n        \"string\",\n        \"null\"\n      ],\n      \"description\": \"The transcript of the user message, if the input was a voice message\"\n    },\n    \"suggestions\": {\n      \"anyOf\": [\n        {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"text\": {\n                \"type\": \"string\",\n                \"description\": \"The text of the suggestion to show to the user\"\n              },\n              \"type\": {\n                \"type\": \"string\",\n                \"enum\": [\n                  \"full\",\n                  \"partial\",\n                  \"action\"\n                ],\n                \"description\": \"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"\n              },\n              \"action\": {\n                \"type\": \"string\",\n                \"enum\": [\n                  \"question\",\n                  \"update\",\n                  \"add\",\n                  \"remove\",\n                  \"breakdown\"\n                ],\n                \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n              }\n            },\n            \"required\": [\n              \"text\",\n              \"type\",\n              \"action\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"description\": \"A set of flow up suggestions for the reply\"\n        },\n        {\n          \"type\": \"null\"\n        }\n      ],\n      \"description\": \"A set of flow up suggestions for the reply\"\n    },\n    \"taskId\": {\n      \"type\": [\n        \"string\",\n        \"null\"\n      ],\n      \"description\": \"The task ID the reply type is associated with, or none if not applicable\"\n    }\n  },\n  \"required\": [\n    \"type\",\n    \"text\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}",
                "exception.stacktrace": "Error: FAILED_PRECONDITION: Generation resulted in no candidates matching provided output schema.\n\nCandidate[0] Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- (root): must be object\n\nProvided data:\n\nnull\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"type\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"question\",\n        \"update\",\n        \"add\",\n        \"remove\",\n        \"breakdown\"\n      ],\n      \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n    },\n    \"text\": {\n      \"type\": \"string\",\n      \"description\": \"The reply to the user message\"\n    },\n    \"transcript\": {\n      \"type\": [\n        \"string\",\n        \"null\"\n      ],\n      \"description\": \"The transcript of the user message, if the input was a voice message\"\n    },\n    \"suggestions\": {\n      \"anyOf\": [\n        {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"text\": {\n                \"type\": \"string\",\n                \"description\": \"The text of the suggestion to show to the user\"\n              },\n              \"type\": {\n                \"type\": \"string\",\n                \"enum\": [\n                  \"full\",\n                  \"partial\",\n                  \"action\"\n                ],\n                \"description\": \"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"\n              },\n              \"action\": {\n                \"type\": \"string\",\n                \"enum\": [\n                  \"question\",\n                  \"update\",\n                  \"add\",\n                  \"remove\",\n                  \"breakdown\"\n                ],\n                \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n              }\n            },\n            \"required\": [\n              \"text\",\n              \"type\",\n              \"action\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"description\": \"A set of flow up suggestions for the reply\"\n        },\n        {\n          \"type\": \"null\"\n        }\n      ],\n      \"description\": \"A set of flow up suggestions for the reply\"\n    },\n    \"taskId\": {\n      \"type\": [\n        \"string\",\n        \"null\"\n      ],\n      \"description\": \"The task ID the reply type is associated with, or none if not applicable\"\n    }\n  },\n  \"required\": [\n    \"type\",\n    \"text\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}\n    at /Users/marcel/Dev/GitHub/mumble-sort/functions/node_modules/@genkit-ai/ai/lib/generate.js:563:15\n    at Generator.next (<anonymous>)\n    at fulfilled (/Users/marcel/Dev/GitHub/mumble-sort/functions/node_modules/@genkit-ai/ai/lib/generate.js:43:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
              },
              "description": "exception"
            }
          }
        ]
      }
    }
  }
}
robinfischerinnogames commented 1 month ago

Jep, I also have this problem. It seems the tool response is validated against the final output schema of the generate call. Tool Requests should be ignored when validating the schema. Or even better: Tool Requests should be validated against the output schema of the tool