[Bug]: Support for multi-turn function calling

[TaoChenOSU]

File Name

gemini/function-calling/function_calling_data_structures.ipynb

What happened?

Hi!

I am experiencing function calling with Gemini. On this document, it says Gemini support multi-turn function calling: https://ai.google.dev/gemini-api/docs/function-calling?_gl=1*5vmwdb*_up*MQ..*_ga*MTk5OTQ2NDQ4Ni4xNzI0OTYzMDg1*_ga_P1DBVKWT6V*MTcyNDk2MzA4NS4xLjAuMTcyNDk2MzA4OS4wLjAuMTI0MjY5MDE1Ng..#function-calling-one-and-a-half-turn-curl-sample. However, in my experience, it doesn't support the following scenario:

• I have two simple functions: one that doesn't take any parameter and just returns the current location of the user (getLocation); another one that take a location and return the weather of that location (getWeather).
• I ask the following question: "What is the weather in my location?". The model is unable to first return a function call to getLocation then wait for the result of the call and finally issue a second turn that asks for getWeather.

Is this something that the model supports?

Relevant log output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[koverholt]

Hi @TaoChenOSU, thanks for the report!

The sample notebook that you mentioned is for the Gemini API on Vertex AI, whereas the docs that you linked to are related to the Gemini API on Google AI.

Are you running that sample notebook with the Vertex AI SDK, or are you running the notebook (or a different code sample) using the Google AI SDK from https://ai.google.dev/gemini-api/docs/function-calling?

Both flavors of the Gemini API (Vertex AI Studio and Google AI Studio) support parallel function calling and multi-turn function calling. So the scenario that you described should work, and is demonstrated using Gemini on Vertex AI in the "Chat example: Using Function Calling in a chat session to answer user's questions about the Google Store" section in the Intro to Function Calling notebook.