[JS] VertexAI.ClientError when providing context from retriever and Gemini calling a provided tool

[jeff-treuting]

Describe the bug When calling generate with an array of available tools and providing the context parameter the results of a Firestore Retriever call, the call errors out with the following error:

{"severity":"ERROR","message":"Unhandled error Error: Vertex response generation failed: ClientError: [VertexAI.ClientError]: Within a single message, FunctionResponse cannot be mixed with other type of part in the request for sending chat message.\n    at /xxxx/node_modules/@genkit-ai/vertexai/lib/gemini.js:514:17\n    at Generator.throw (<anonymous>)\n    at rejected (/xxxx/node_modules/@genkit-ai/vertexai/lib/gemini.js:50:29)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"}

To Reproduce Here is some simplified code that shows the setup that produces the error for me every time. When I comment out passing in the context, it works as expected.

const myTool = defineTool(
  {
    name: "myTool",
    description:
      "When a question about the calendar bookings or reservations is asked, this tool will load the calendar booking data.",
    inputSchema: z.object({
      beginDate: z
        .string()
        .optional()
        .describe("The beginning date to load the calendar data for."),
      endDate: z
        .string()
        .optional()
        .describe("The end date to load the calendar data for."),
      member: z
        .string()
        .optional()
        .describe("The member ID or name to filter the calendar bookings by."),
    }),
    outputSchema: z.string(),
  },
  async (input) => "haha Just kidding no joke about for you! got you"
);

    const myRetrieverRef = defineFirestoreRetriever({
      name: "retriever-id",
      firestore: getFirestore(),
      collection: "calendar/vectorIndex",
      contentField: "content",
      vectorField: "embedding",
      embedder: textEmbeddingGecko,
      distanceMeasure: "COSINE", // 'EUCLIDEAN', 'DOT_PRODUCT', or 'COSINE' (default)
    });

    const docs = await retrieve({
      retriever: myRetrieverRef,
      query: userMessage,
      options: {
        limit: 5,
        k: 3,
      },
    });

    const myPrompt = await prompt("myBot");
    const result = await myPrompt.generate({
      input: {
        message: userMessage,
        today: format(new Date(), "yyyy-MM-dd"), // "2022-01-01
        user: {
          id: request.auth?.uid || "anonymous",
          name: request.auth?.token.name || "anonymous",
        },
      },
      context: docs, // when I comment out this line the error goes away regardless of what tool I include, I've tried a few different versions of it
      tools: [myTool],
    });

Expected behavior I would expect to be able to get results that utilize a cusotm tool for retrieving information form a data source and provide context from a retriever as well.

Runtime (please complete the following information):

• OS: MacOS
• Version: Sonoma 14.4.1 ** Node version
• v20.11.1

[ceit-each]

I am hitting the same issue.

[jeff-treuting]

It appears the issue is form Gemini. Here is the place in the gemini code where the error message comes from and seems like it is not supported at this point:

https://github.com/google-gemini/generative-ai-js/blob/5739e2a32736a2c8b1e2689e0d68dd3565b3baae/packages/main/src/requests/request-helpers.ts#L95

[cabljac]

Hi, just to clarify, is this using the Vertex AI plugin? The package you linked is from Google AI, but the title of the issue mentions Vertex.

Could you provide your configureGenkit options? (redacting anything sensitive of course)