search

CsabaConsulting / InspectorGadgetApp

Open Multi-Modal Personal Assistant
MIT License
4 stars 1 forks source link

Gemini 1.5 Pro results in 500 Internal error with as many tools as Flash #12

Closed MrCsabaToth closed 2 months ago

MrCsabaToth commented 4 months ago

Even though I disabled some tools for gemini-1.5-flash (the Exchange Tool and either the Web Search (Duck Duck Go) or the Tavily Web research tool, and also the Alpha Vantage), but Flash was working with several tools. It seems Pro cannot take as many, or we need to tune some function descriptions? The 500 error happens during the model initialization.

MrCsabaToth commented 4 months ago

For MVP sprint we can neglect this IMHO and roll with the Flash. We can figure it out after the hackathon deadline.

MrCsabaToth commented 3 months ago

There's an idea to stuff the prompt with the list of the available tools and functions. Even though passing the available tools in a native way should give the information to the model, but maybe reinforcing that with a repeated own version would help. Joan Boronat Ruiz advised that and I'll try it https://medium.com/@tocsa2/i-dont-think-it-s-a-good-idea-to-repeat-the-tools-in-the-system-prompt-because-the-model-receives-e90a7df72859

MrCsabaToth commented 3 months ago

I'm using the extra function stuffing. It didn't prevent some internal errors, however it seems to not interfere with the native tool list because I was able to make queries which require two rounds of function calls (weather prompt uses the location function first and then the weather function).

MrCsabaToth commented 2 months ago

Another related issue https://github.com/google-gemini/generative-ai-dart/issues/194

MrCsabaToth commented 2 months ago

We are now converting over to https://pub.dev/packages/firebase_vertexai (#53), let's see how that deals with functions. So far not good (Unable to submit request because function parameters schema should be of type OBJECT. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling), the functions worked with the other package already, we are a step back again.

MrCsabaToth commented 2 months ago

Not 500, but after transition we are dealing with other issues, but not 500s