Tool is called repeatedly

[benwager]

I am trying to create a node console application, but...

I am getting multiple (sometimes 10+) calls being made to my tool function calls.

Even when using the example get_weather as per docs:

import dotenv from 'dotenv';
import { RealtimeClient } from '@openai/realtime-api-beta';
dotenv.config();

const client = new RealtimeClient({ apiKey: process.env.OPENAI_API_KEY });

client.on('error', (event) => {
  console.log(`Error: ${event}`);
});

client.on('conversation.item.completed', ({ item }) => {
  if (item.type === 'function_call') {
    console.log("Function call");
    console.log(item.name);
    console.log(item.arguments);
  }
  else if (item.type === 'function_call_output') {
    console.log("Function completed");
    console.log(item.output);
  }
});

client.addTool(
  {
    name: 'get_weather',
    description:
      'Retrieves the weather for a given lat, lng coordinate pair. Specify a label for the location.',
    parameters: {
      type: 'object',
      properties: {
        lat: {
          type: 'number',
          description: 'Latitude',
        },
        lng: {
          type: 'number',
          description: 'Longitude',
        },
        location: {
          type: 'string',
          description: 'Name of the location',
        },
      },
      required: ['lat', 'lng', 'location'],
    },
  },
  async ({ lat, lng, location }) => {
    const result = await fetch(
      `https://api.open-meteo.com/v1/forecast?latitude=${lat}&longitude=${lng}&current=temperature_2m,wind_speed_10m`,
    );
    const json = await result.json();
    return json;
  },
);

client.updateSession({
  instructions: "You are a helphful assistant",
  tool_choice: 'required',
  modalities: ["text"]
});

await client.connect();
console.log("connected to realtime API")

client.sendUserMessageContent([{ type: 'input_text', text: `how's the weather in amsterdam?` }]);

The output of console:

connected to realtime API
Function call
get_weather
{"lat":52.3676,"lng":4.9041,"location":"Amsterdam"}
Function completed
{"latitude":52.366,"longitude":4.901,"generationtime_ms":0.016927719116210938,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":5,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-09T14:30","interval":900,"temperature_2m":15.2,"wind_speed_10m":13.7}}
Function call
get_weather
{"lat":52.3676,"lng":4.9041,"location":"Amsterdam"}
Function completed
{"latitude":52.366,"longitude":4.901,"generationtime_ms":0.01895427703857422,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":5,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-09T14:30","interval":900,"temperature_2m":15.2,"wind_speed_10m":13.7}}
Function call
get_weather
{"lat":52.3676,"lng":4.9041,"location":"Amsterdam"}
Function completed
{"latitude":52.366,"longitude":4.901,"generationtime_ms":0.028967857360839844,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":5,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-09T14:30","interval":900,"temperature_2m":15.2,"wind_speed_10m":13.7}}

It usually hits it more than this, but I kill it early

[khorwood-openai]

Can you log the item.ids as well? Wondering whether this is a model issue or a library issue. Is it possible to replicate with a test case?

[zakir0101]

I think the problem is with this line : tool_choice: 'required', Cause at the beginning, the model will send a function call to query wether status, which is desired. But after that when the function response, is submitted to the model to create a second response ( which happen automatically as mentioned in Readme ) ... At this point the model should logically not return a function call again , and Normally it should extract the desired data from the response and rephrase them in a readable format. But I assume, since tool choice is set to required, the model will be force to call the function again, though it doesn't make sense hier

[zakir0101]

One solution is to update the session hier: const result = await fetch( https://api.open-meteo.com/v1/forecast?latitude=${lat}&longitude=${lng}&current=temperature_2m,wind_speed_10m, ); const json = await result.json(); // ToDo : write code hier to update the session making tool choice NOT required return json

[benwager]

What @zakir0101 said seems to be correct. As soon as I remove tool_choice: 'required', it only triggers a single function call.

So for now, it seems best to follow @zakir0101 suggestion to update session and remove tool_choice required option after the first function call completion. And then add it again for subsequent requests.

@khorwood-openai

Here is output with the item.id logged on both function_call and function_call_output

connected to realtime API
Function call
get_weather
{"lat":52.3676,"lng":4.9041,"location":"Amsterdam"}
item_AGhf3owGwo3zIDI8J2jIh
Function completed
{"latitude":52.366,"longitude":4.901,"generationtime_ms":0.03600120544433594,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":5,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-10T07:00","interval":900,"temperature_2m":13.5,"wind_speed_10m":20.9}}
item_AGhf4R6ikbkJUvJ4fbDlO
Function call
get_weather
{"lat":52.3676,"lng":4.9041,"location":"Amsterdam"}
item_AGhf4C1Xf0qsiSPR9Zoqz
Function completed
{"latitude":52.366,"longitude":4.901,"generationtime_ms":0.024080276489257812,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":5,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-10T07:00","interval":900,"temperature_2m":13.5,"wind_speed_10m":20.9}}
item_AGhf4OZxhEoHS8f5Bfs8w
Function call
get_weather
{"lat":52.3676,"lng":4.9041,"location":"Amsterdam"}
item_AGhf4R71Ti0WKjRILxskY