Function calling only considers functions in the first tool?

[MrCsabaToth]

Description of the bug:

Using gemini-1.5-flash. Prompt: "What is the weather today?"

Tool configuration 1:

[
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchWeatherForecast',
        'Returns the weather in a given location.',
        Schema(
          SchemaType.object,
          properties: {
            'latitude': Schema.number(
              description: 'Latitude of the weather observation and forecast',
            ),
            'longitude': Schema.number(
              description:
              'Longitude of the weather observation and forecast',
            ),
          },
          requiredProperties: ['latitude', 'longitude'],
        ),
      ),
    ],
  ),
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchGpsLocation',
        'Returns the latitude and longitude of the current GPS location.',
        Schema(SchemaType.string),
      ),
      FunctionDeclaration(
        'fetchHeartRate',
        'Returns the current heart rate measurement.',
        Schema(SchemaType.integer),
      ),
    ],
  ),
]

Response 1: I need to know your location to get the weather forecast. Can you tell me your latitude and longitude?

Tool configuration 2 (switching the order of the two tools):

[
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchGpsLocation',
        'Returns the latitude and longitude of the current GPS location.',
        Schema(SchemaType.string),
      ),
      FunctionDeclaration(
        'fetchHeartRate',
        'Returns the current heart rate measurement.',
        Schema(SchemaType.integer),
      ),
    ],
  ),
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchWeatherForecast',
        'Returns the weather in a given location.',
        Schema(
          SchemaType.object,
          properties: {
            'latitude': Schema.number(
              description: 'Latitude of the weather observation and forecast',
            ),
            'longitude': Schema.number(
              description:
              'Longitude of the weather observation and forecast',
            ),
          },
          requiredProperties: ['latitude', 'longitude'],
        ),
      ),
    ],
  ),
]

Response 2: I am sorry, I cannot fulfill this request. I do not have access to weather information.

Artificially unifying the two tools into a virtual one, merging all functions under that:

[
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchWeatherForecast',
        'Returns the weather in a given location.',
        Schema(
          SchemaType.object,
          properties: {
            'latitude': Schema.number(
              description: 'Latitude of the weather observation and forecast',
            ),
            'longitude': Schema.number(
              description:
              'Longitude of the weather observation and forecast',
            ),
          },
          requiredProperties: ['latitude', 'longitude'],
        ),
      ),
      FunctionDeclaration(
        'fetchGpsLocation',
        'Returns the latitude and longitude of the current GPS location.',
        Schema(SchemaType.string),
      ),
      FunctionDeclaration(
        'fetchHeartRate',
        'Returns the current heart rate measurement.',
        Schema(SchemaType.integer),
      ),
    ],
  ),
]

The model properly asks first for the location tool for the lat / lon coordinates, and then with a second round of call (where I pass the location function call result down) it properly asks to call the weather function with the proper GPS coordinates.

Respnose: The weather today is clear and the temperature is 25 degrees Celsius. The wind is blowing from the southwest at 2 meters per second.

Actual vs expected behavior:

I'd expect to be able to keep the two tools apart for extensible software architecture. So far I'm aiming for 8 tools, many of those have multiple functions.

Any other information you'd like to share?

I peeked at the code and it looks to me that ultimately a REST API call is made. The _generateContentRequest seems to serialize the tools OK (by looking at it) if (tools != null) 'tools': tools.map((t) => t.toJson()).toList(), https://github.com/google-gemini/generative-ai-dart/blob/8a58d778e2ef254901e3319f238b147788211ecf/pkgs/google_generative_ai/lib/src/model.dart#L333 so I don't know yet where's the problem. Maybe in my code?

[MrCsabaToth]

Correction: this is with Gemini 1.5 Flash, I'll experiment with Pro again, possibly file another issue (cannot deal with more than 8 functions)

[MrCsabaToth]

Something which could affect this bug: while I was converting over the code to use https://pub.dev/packages/firebase_vertexai I got the error Unable to submit request because function parameters schema should be of type OBJECT. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling. The two affected functions indeed have non object SchemaType, because the heart rate is an integer (SchemaType.integer) and I declare the location as a SchemaType.string which would just contain the two coordinates. And neither of the functions have any input parameters.