google-gemini / generative-ai-swift

The official Swift library for the Google Gemini API
https://ai.google.dev/gemini-api/docs/get-started/tutorial?lang=swift
Apache License 2.0
922 stars 148 forks source link

Function calling with a response mime type: 'application/json' is unsupported #195

Closed skoru01 closed 1 month ago

skoru01 commented 3 months ago

Description of the bug:

Hi, i am using Function calling along with generationConfig and then i am getting below error message.

How can we use function calling tools along with GenerationConfig in this SDK, since i want to add temperature, topK and topP to function calling for more accuracy response.

Request:

Screenshot 2024-07-29 at 12 38 26 PM
Response payload: {
  "error": {
    "code": 400,
    "message": "Function calling with a response mime type: 'application/json' is unsupported",
    "status": "INVALID_ARGUMENT"
  }
}

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

andrewheard commented 3 months ago

Hi @skoru01, this error is specifically for responseMIMEType in GenerationConfig. It's possible to set temperature, topK and topP with function calling, for example:

let generativeModel = GenerativeModel(
  name: "gemini-1.5-flash-latest",
  apiKey: apiKey,
  generationConfig: GenerationConfig(temperature: 0.0, topP: 0.99, topK: 1),
  tools: [Tool(functionDeclarations: [...])]
)

If you'd like to use JSON format responses in some cases, and function calling in others, you could create a separate instance of the model / generation config for the two use cases. Does that answer your question?

skoru01 commented 3 months ago

Hi @andrewheard Thanks for the suggestion. i will try it. Is it not possible to use responseMIMEType along with function calling? function calling is giving more accurate options and i want json as well.

i am using gemini-1.5-flash-latest and sometimes i am getting pretty python or json string instead of JSON. Where as gemini-1.5-pro-latest really not giving accurate result as like gemini-1.5-flash-latest

andrewheard commented 3 months ago

@skoru01 Would it be possible for you to describe your use case of function calling in more detail? Potentially you could add an additional function for your final response and use the function calling mode ANY. This way it would never produce a natural-language/text response. In Swift the setup would look something like:

let model = GenerativeModel(
  ...,
  tools: [Tool(functionDeclarations: [...])],
  toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: .any))
)
skoru01 commented 3 months ago

@andrewheard Thanks you, i will check this and get back to you if its not working.

cabljac commented 3 months ago

Hi! I'm looking an issue over in genkit and I think it's related/the same problem, but in Node.js. We're setting responseMimeType as 'application/json' and also adding tools in a chat request. Is this not possible then?

@andrewheard

github-actions[bot] commented 2 months ago

Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs.

skoru01 commented 2 months ago

@skoru01 Would it be possible for you to describe your use case of function calling in more detail? Potentially you could add an additional function for your final response and use the function calling mode ANY. This way it would never produce a natural-language/text response. In Swift the setup would look something like:

let model = GenerativeModel(
  ...,
  tools: [Tool(functionDeclarations: [...])],
  toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: .any))
)

Hi @andrewheard Sorry for the late response. GenerationConfig working good. but below code causing issue

let model = GenerativeModel(
  ...,
  tools: [Tool(functionDeclarations: [...])],
  toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: .any))
)

Response

{
  "error": {
    "code": 500,
    "message": "An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting",
    "status": "INTERNAL"
  }
}

FYI, if send .auto over FunctionCallingConfig, its working fine.

github-actions[bot] commented 1 month ago

Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs.

github-actions[bot] commented 1 month ago

This issue was closed because it has been inactive for 28 days. Please post a new issue if you need further assistance. Thanks!