[BUG] Issue with tool calling when using relay server.

marcelvanworkum commented 1 week ago

I noticed an issue with the current stateful design of the RealtimeClient when using addTool.

Error

The frontend RealtimeClient calls addTool, updating the internal state to know that those tools are available. However, in our relay server, we don't call addTool.

This results in errors being thrown from the relay server's RealtimeClient instance: "Tool ... has not been added". And causes strange behaviour where the AI will say that it doesn't have access to that tool right now, but in the frontend the tool actually still ends up running just fine.

Screenshot 2024-11-06 at 4 56 49 PM

Exploration

This is actually what you'd expect to happen, as these tools are (at least from the perspective of the relay server) not added.

However, the frontend actually ends up receiving the tool call message anyway and running the function. So visually the tool call works (although you do see the error in the logs).

If we look at the tool_call function within the lib:

    const callTool = async (tool) => {
      try {
        const jsonArguments = JSON.parse(tool.arguments);
        const toolConfig = this.tools[tool.name];
        if (!toolConfig) {
          throw new Error(`Tool "${tool.name}" has not been added`);
        }
        const result = await toolConfig.handler(jsonArguments);
        this.realtime.send('conversation.item.create', {
          item: {
            type: 'function_call_output',
            call_id: tool.call_id,
            output: JSON.stringify(result),
          },
        });
      } catch (e) {
        this.realtime.send('conversation.item.create', {
          item: {
            type: 'function_call_output',
            call_id: tool.call_id,
            output: JSON.stringify({ error: e.message }),
          },
        });
      }
      this.createResponse();
    };

We see that in the exception case we are creating a new conversation item:

      } catch (e) {
        this.realtime.send('conversation.item.create', {
          item: {
            type: 'function_call_output',
            call_id: tool.call_id,
            output: JSON.stringify({ error: e.message }),
          },
        });
      }

This means that frontend 1; receiving the tool call response, but 2; receiving this error response in the conversation, which results in the AI telling us that it's "unable to find the weather in X location right now".

Resolution

A work around for this is to simply call addTool in the relay server as well. But obviously those functions will do nothing there as they are client-side.

Another potential solution I explored is to call updateSession directly, and pass in the tool definitions. But again callTool is expecting those tools to be added via addTool and be in the RealtimeClient's state:

const toolConfig = this.tools[tool.name];
if (!toolConfig) {
  throw new Error(`Tool "${tool.name}" has not been added`);
}

It seems like we'd want to be able to define the tools we want to use on the client side, but still have the server side instance of RealtimeClient be able to access those. Curious to hear if there is some more correct way to achieve this behaviour, or if this is a limitation with the current API design.

Also just want to say that I am very impressed with the real time api, and really do appreciate you folks taking the time to put together this demo app for all of us :)

marcelvanworkum commented 1 week ago

Related issues:

marcelvanworkum commented 1 week ago

Based on the suggestion here I've created a pr which switches the relay server from the RealtimeClient to just using the RealtimeAPI. This fixes the issue of "Tool ... has not been added."

Although I have not done enough testing to see if it breaks anything else downstream that relies on the stateful RealtimeClient.

marcelvanworkum commented 1 week ago

Potentially solved by https://github.com/openai/openai-realtime-api-beta/pull/53

youens commented 4 hours ago

This fixed the problem for me. 👍 Thanks @marcelvanworkum!

openai / openai-realtime-console