Azure-Samples / aoai-realtime-audio-sdk

Azure OpenAI code resources for using gpt-4o-realtime capabilities.
MIT License
663 stars 117 forks source link

Can we have an example of adding tools? Such as simply querying the weather. #39

Open skxgood03 opened 1 month ago

skxgood03 commented 1 month ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful


Thanks! We'll be in touch soon.

skxgood03 commented 1 month ago

`function createConfigMessage(): SessionUpdateMessage {

let configMessage: SessionUpdateMessage = {
    type: "session.update",
    session: {
        turn_detection: {
            type: "server_vad",
        },
        input_audio_transcription: {
            model: "whisper-1"
        },
        tools: [
            {
                type: "function",
                name: "get_weather_for_location",
                description: "gets the weather for a location",
                parameters: {
                    type: "object",
                    properties: {
                        location: {
                            type: "string",
                            description: "The city and state e.g. San Francisco, CA"
                        },
                        unit: {
                            type: "string",
                            enum: ["c", "f"]
                        }
                    },
                    required: ["location", "unit"]
                }
            }
        ],
        tool_choice: {
            type: "function",
            function: "get_weather_for_location"
        }
    }
};

const systemMessage = getSystemMessage();
const temperature = getTemperature();
const voice = getVoice();

if (systemMessage) {
    configMessage.session.instructions = systemMessage;
}
if (!isNaN(temperature)) {
    configMessage.session.temperature = temperature;
}
if (voice) {
    configMessage.session.voice = voice;
}

return configMessage;

}`I added tools here, but I don't understand how to write it later to make it complete.

DemacL commented 1 month ago
  1. case 'response.function_call_arguments.done':
        handleFunctionCall(message)
        break

2.

  async function handleFunctionCall(message: any) {

    params = JSON.parse(message.arguments)

    let funcName = message.name
    if (funcName === 'XXXX') {
        console.log('funcName', funcName) //  get Data
        const data = 'your data'

        const msg = {
            type: "conversation.item.create",
            previous_item_id: message.item_id,
            item: {
                type: 'message',
                role: "user",
                content: [{
                    type: "input_text",
                    text: `$ {
                        handle your data
                    }`,
                }],
                status: 'completed',
            }
        }
        realtimeStreaming.send(msg)
    }

    realtimeStreaming.send({
        type: "response.create"
    })
}
skxgood03 commented 1 month ago
   case 'response.function_call_arguments.done':
        handleFunctionCall(message)
        break
  async function handleFunctionCall(message: any) {

    params = JSON.parse(message.arguments)

    let funcName = message.name
    if (funcName === 'XXXX') {
        console.log('funcName', funcName) //  get Data
        const data = 'your data'

        const msg = {
            type: "conversation.item.create",
            previous_item_id: message.item_id,
            item: {
                type: 'message',
                role: "user",
                content: [{
                    type: "input_text",
                    text: `$ {
                        handle your data
                    }`,
                }],
                status: 'completed',
            }
        }
        realtimeStreaming.send(msg)
    }

    realtimeStreaming.send({
        type: "response.create"
    })
}

Thank you, I have been solved.