search

google-gemini / generative-ai-dart

The official Dart library for the Google Gemini API
https://ai.google.dev/gemini-api/docs/get-started/tutorial?lang=dart
Apache License 2.0
551 stars 114 forks source link

Is Function Calling possible? #81

Open jaydanurwin opened 7 months ago

jaydanurwin commented 7 months ago

I've looked into the samples a bit and the available params but I'm not seeing anything about function calling/tools. Is this already possible and I'm missing it or is it planned to be supported? It would be awesome to be able to call local Dart functions through Gemini similar to how it appears to be possible with Python or a REST API

MrCsabaToth commented 3 months ago

Badly needed

davidmigloz commented 3 months ago

It's already supported. https://ai.google.dev/gemini-api/docs/function-calling/tutorial?lang=dart

MrCsabaToth commented 3 months ago

That'd be dope! I'll try that next week!

adamglin0 commented 3 months ago

It's already supported. https://ai.google.dev/gemini-api/docs/function-calling/tutorial?lang=dart

I try function call by this post, but is not works. Gemini return this: 

I need to access a live exchange rate API to answer your question. I do not have access to real-time information, including currency exchange rates.  Therefore, I cannot provide you with the current exchange rate for US dollars to Swedish krona. 

To get the most up-to-date information, I recommend checking a reliable online converter or your bank's website.
MrCsabaToth commented 3 months ago

I try function call by this post, but is not works. Gemini return this:

I need to access a live exchange rate API to answer your question. I do not have access to real-time information, including currency exchange rates.  Therefore, I cannot provide you with the current exchange rate for US dollars to Swedish krona. 

To get the most up-to-date information, I recommend checking a reliable online converter or your bank's website.

What's the code you are configuring the function / tool with, and then calling Gemini?

adamglin0 commented 3 months ago

@MrCsabaToth

Sorry, if I set model as 'gemini-1.5-flash' it works!

MrCsabaToth commented 3 months ago

@MrCsabaToth

Sorry, if I set model as 'gemini-1.5-flash' it works!

No problem but I'm still highly interested in the source code

MrCsabaToth commented 2 months ago

Actually now I'm at the point that I'm trying function calling and it doesn't seem to be working. I was providing a function to retrieve weather data. I tried both gemini-1.5-flash-latest and gemini-1.5-pro-latest with the prompt "What is the weather today?", and I got back I am sorry, I cannot provide you with the weather information. I do not have access to real-time data like weather conditions.. Note that I also provide a tool to retrieve the location.

adamglin0 commented 2 months ago

Actually now I'm at the point that I'm trying function calling and it doesn't seem to be working. I was providing a function to retrieve weather data. I tried both gemini-flash-lates and gemini-pro-latest with the prompt "What is the weather today?", and I got back I am sorry, I cannot provide you with the weather information. I do not have access to real-time data like weather conditions.. Note that I also provide a tool to retrieve the location.

Do you add function_calling_config parameter? Try to delete this.

MrCsabaToth commented 2 months ago

Actually now I'm at the point that I'm trying function calling and it doesn't seem to be working. I was providing a function to retrieve weather data. I tried both gemini-flash-lates and gemini-pro-latest with the prompt "What is the weather today?", and I got back I am sorry, I cannot provide you with the weather information. I do not have access to real-time data like weather conditions.. Note that I also provide a tool to retrieve the location.

Do you add function_calling_config parameter? Try to delete this.

No, do you have some code by any chance? function_calling_config seems to be Pythonic, we are in Dart land here. However the Dart API also has a FunctionCallingMode. It has modes: auto, any, and none. any forces a function call, auto supposed to be the default. I'll try to add a toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: FunctionCallingMode.auto)) during model instantiation. Unlike the Python and some other language APIs the chat instantiation doesn't have a parameter like start_chat(enable_automatic_function_calling=True) in Python. The Dart startChat only has history, safetySettings, and generationCOnfig. This latter doesn't have function call related settings in it.

@adamglin0 How did you solve the problem?

MrCsabaToth commented 2 months ago

I tested explicitly specifying toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: FunctionCallingMode.auto)) and it didn't help. My source code: https://github.com/CsabaConsulting/InspectorGadgetApp/blob/21cfcc45436dd1c155c703183d22c0239376ee56/lib/interaction/view/interaction_page.dart#L257

adamglin0 commented 2 months ago

Actually now I'm at the point that I'm trying function calling and it doesn't seem to be working. I was providing a function to retrieve weather data. I tried both gemini-flash-lates and gemini-pro-latest with the prompt "What is the weather today?", and I got back I am sorry, I cannot provide you with the weather information. I do not have access to real-time data like weather conditions.. Note that I also provide a tool to retrieve the location.

Do you add function_calling_config parameter? Try to delete this.

No, do you have some code by any chance? function_calling_config seems to be Pythonic, we are in Dart land here. However the Dart API also has a FunctionCallingMode. It has modes: auto, any, and none. any forces a function call, auto supposed to be the default. I'll try to add a toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: FunctionCallingMode.auto)) during model instantiation. Unlike the Python and some other language APIs the chat instantiation doesn't have a parameter like start_chat(enable_automatic_function_calling=True) in Python. The Dart startChat only has history, safetySettings, and generationCOnfig. This latter doesn't have function call related settings in it.

@adamglin0 How did you solve the problem?

Let me find my old code, this will take some time.

adamglin0 commented 2 months ago

@MrCsabaToth I can no longer reproduce the issue. But I can tell you how I resolved it.

  1. After I found the same error, I tried this document.

But still not successful.

  1. Then I found that my GenerativeModel section seemed different from the one in the document, so I copied part of the model and it worked. https://ai.google.dev/gemini-api/docs/function-calling/tutorial?hl=zh-cn&lang=dart

  2. Later, after leaving a message on this issue, I modified the value of the model, but then realized that I wrote toolConfig in GenerativeModel (it seems like this because there is no git record).

  3. Now I cannot reproduce it anymore, sorry for that

MrCsabaToth commented 2 months ago

Reporting back about what I've found out: @adamglin0 was exactly right here https://github.com/google-gemini/generative-ai-dart/issues/81#issuecomment-2172053459

If I go with gemini-1.5-flash or gemini-1.5-pro instead of gemini-1.5-flash-latest or gemini-1.5-pro-latest (so not using the -latest model name prefix) the tool functionality seems to be waking up. Now I'm getting I need to know your location to get the weather forecast. Can you tell me your latitude and longitude?. So it is probably seeing my weather function, however it misses the location function which would exactly provide the lat / long.

Since https://github.com/google-gemini/generative-ai-dart/blob/main/samples/dart/bin/function_calling.dart demonstrates a function call chain possibility, I expect the model to figure out the two step of first calling the location function for the lat long GPS, and then use those as parameters to call the weather function.

MrCsabaToth commented 2 months ago

I'm dealing with a problem with multiple tools: #194