Add ToolConfig to force tool calls (currently in apiVersion v1beta1)

[jemc]

Is your feature request related to a problem? Please describe. I'm frustrated when Gemini responds in natural language rather than a tool call I wanted to force it to use.

VertexAI v1beta1 has ToolConfig, which allows for forcing the model to make a tool call, or even call one of a specific set of allowed tools.

Describe the solution you'd like Add tool_config: ToolConfig to GenerateContentRequest, and update the generateContent function to switch to v1beta1 as the apiVersion when the tool_config option is present, similar to how this library used to switch the apiVersion based on whether the tools options was present

Describe alternatives you've considered Alternatively, you could allow the caller to specify a different apiVersion explicitly, which would allow the caller to use any v1beta1 features that you haven't yet added support for in this library.

Currently I am monkey-patching the library to make this happen within my application.

Additional context For my use cases, forced tool calling is a hard requirement, and without the option to force it, I can't use Gemini.

[thegrandpoobah]

@jemc would you happen to have a gist/whathaveyou of how you've monkey patched the library? i'm starting down a similar path and it would be useful reference if you are willing to share.

[gtanczyk]

I have encountered this limitation, and for me this way of monkey patching worked:

(works for me, but I'm using only function calling for the moment)

[dtinth]

For others who stumble upon this issue, it turns out that the @google-cloud/vertexai npm package is very much outdated from the latest capabilities...

But thankfully, @google-cloud/aiplatform also exists with up-to-date definitions. We can use that instead, to access the latest capabilities.

However, it is quite under-documented in the guides section, but the API reference has the complete documentation there. Here’s an example how to use it:

const projectId = "________";
const location = "________";
const publisher = "google";
const modelName = "gemini-1.5-pro-001";

import * as aiplatform from "@google-cloud/aiplatform";

const client = new aiplatform.v1beta1.PredictionServiceClient({
  // `fallback` must be set to true to this to work, I don't know why...
  fallback: true,
});

const [result] = await client.generateContent({
  model:
    `projects/${projectId}/locations/${location}/publishers/${publisher}/models/${modelName}`,
  contents: [{ role: "user", parts: [{ text: "________" }] }],
  toolConfig: {
    functionCallingConfig: {
      mode: "ANY",
    },
  },
  tools: [
    {
      functionDeclarations: [
        /* ________ */
      ],
    },
  ],
});

console.log(JSON.stringify(result, null, 2));

[happy-qiao]

ToolConfig is added at https://github.com/googleapis/nodejs-vertexai/releases/tag/v1.6.0