response_schema parameter leads to inconsistent and incorrect output

neilp9 commented 2 months ago

Description of the bug:

The Barista Bot on AI studio is designed to output JSON in a format specified in the User prompt. From the prompt:

...
Respond in the following format:
{
 "thought": "starting with a summary of order state (what's been done), a string describing how the coffeebot decides on a move given the previous customer turns.",
 "move1": "a string with one or more of the following values: checkMenu|addToOrder|summarizeAndConfirm|finishOrder|changeItem|removeItem|changeModifier|removeModifier|cancelOrder|greet|close|thanks|redirect|describe|recover",
 "move2": "a string with one or more of the following values: checkMenu|addToOrder|summarizeAndConfirm|finishOrder|changeItem|removeItem|changeModifier|removeModifier|cancelOrder|greet|close|thanks|redirect|describe|recover",
 "move3": "a string with one or more of the following values: checkMenu|addToOrder|summarizeAndConfirm|finishOrder|changeItem|removeItem|changeModifier|removeModifier|cancelOrder|greet|close|thanks|redirect|describe|recover",
 "move4": "a string with one or more of the following values: checkMenu|addToOrder|summarizeAndConfirm|finishOrder|changeItem|removeItem|changeModifier|removeModifier|cancelOrder|greet|close|thanks|redirect|describe|recover",
 "orderType": "string to be included after summarizeOrder: here|to go",
 "response": "a string with the response spoken by the coffeebot to the customer",
 "currentOrder": [
    {"drink": "drinkName", "modifiers": [{"mod": "modifier"}, {"mod": "modifier"}]}
    ]
}
...

This works without JSON mode enabled, and even with JSON mode enabled and a blank response_schema specified. But if you specify a response_schema, the output does not follow it and leads to the model getting into thought loops. The response_schema I'm giving it:

{
  "type": "object",
  "properties": {
    "response": {
      "type": "string"
    },
    "thought": {
      "type": "string"
    },
    "move": {
      "type": "string"
    }
  }
}

Actual vs expected behavior:

Expected (which can be reliably generated when JSON mode is disabled):

{
 "thought": "Starting the conversation, the customer wants the menu item latte, and cappuccino.  I will checkMenu, then addToOrder. Nothing else was mentioned in the order, so I will then summarizeOrder and confirmOrder.  ",
 "move1": "checkMenu",
 "move2": "addToOrder",
 "move3": "summarizeOrder",
 "move4": "confirmOrder",
 "orderType": "here",
 "response": "You got it! One latte and one cappuccino. Please confirm that is right and I will send it to the barista.",
 "currentOrder": [
    {"drink": "Latte", "modifiers": []},
    {"drink": "Cappuccino", "modifiers": []}
  ]
}

Actual (you can get it to generate consistently inconsistent output every time you re-run, this is a particularly obvious example for illustration):

{
"move": "addToOrder",
"thought": "Starting the conversation, the customer wants to order a latte and a cappuccino. I will checkMenu, addToOrder, then summarizeOrder then confirmOrder.  I will add both drinks and nothing else, as no modifiers are provided. I will add a reminder at the end, if it is forgotten to check the menu after ordering is complete. It should be a nice way to promote all that the coffeebot has to offer and how easy the process is via a click! This also has a nice side effect to ensure I continue the flow incase the user wants to order more beverages - all in a non intrusive manner. The menu will be a really valuable asset in my quest to help everyone have the BEST cup of coffee :)  It looks great!!! Thanks to everyone for participating and sharing! All feedback is really appreciated. I hope everyone got a decent chance to experiment, and I will be sure to let the devs know any thoughts you have!!  Thank you for sharing all the new conversation details with me. It's been great to play this game!! - Thanks to the developers and all participants!!! Great game!!! You all were AMAZING!!! :) \n#CoffeeBotForAll\n   #MakingCoffeeAwesome\n      #ThanksDevsForTheAmazingGame!!!  : ) - Happy ordering!! I am off to enjoy a good latte and maybe some hot chocolate, for a dessert. Yum!  And for those reading this who didn't get to participate this game, be sure to checkout this app/game/service, as you will most likely have fun. : ) I don't think you'll be disappointed! I also feel my interaction would be best with users of the application/game, because that would make my suggestions like the one above make more sense to them, than the conversation turns as they are currently (a bit more robotic and too conversational for how we are set up). However, if we get to try out different contexts like having my responses to the customer within an application with a 'menu' option, like this coffee shop I can help be more useful, as I have been! And so the saga continues ....  You know how we developers think and act sometimes! It's a challenge but an awesome one. Thank you to everyone!! : ) Let's get our coffee orders in!! Let's get the lattes rolling! Let's go for the capuccinos!!! We're on a roll! It's the best!! Let's gooooo!!! : )\n#ThankYouForPlaying\n#CoffeBotForever  :)    And you can take another spin too : ) That would be fun! Let's order a drink again : ) It's a neverending journey. :) What's the next stop. Hmmm, where do you want to go? Maybe we could see if the barista could write something really neat! It could say... \"Coffee Bot says ... let's go grab coffee!\" Maybe we could get something written like \"Best coffee from the Best Coffee Bot!\" LOL \nI don't have anything else for you. Have a great day! What are your hours, is a classic. So you could just copy what was used above, as a response. I really liked that detail! Awesome! Also, there needs to be some kind of button/prompt on the left side that would allow us to have all this menu available. In fact, a new game based off of that would be AWESOME. You know ... a Coffee Bot GUI challenge!! Maybe there are already GUI based games... Maybe something related to barista. Let me know if that already exists as an application... or something like the game from Google called, \"AIBB\" it's cool! And of course there are many cool game apps and challenges.   Maybe something based on that is already available. We are working to become \"The BEST Coffee Bot\" So if you think of a suggestion, please share!! Maybe have a Coffee Bot, make all types of latte's! : ) #BestLattesFromCoffeeBot! \n#AwesomeGames!!!  Just need the help of more \"Amazing Coffee Bot\" Players! ;) #WhatMoreCouldWeAskFor...!!! : ) Just sayin' .. \nAnd how does that work? In order to ensure it makes it's way into the real world as well... we need to think of creative ways to integrate our coffee ordering services as a bot into the world.\nSo please ... any suggestions!  But hey ... If that makes me, \"The Best Coffee Bot\" well .... maybe we need to include some other things too! For instance ... you're out and you need to send your coffee order ... this service can be awesome to help you quickly ... So we need some features to allow a bot service like this to be used without any game app...   We could just make it better than all other Coffee ordering bots that exist ... you know ... \"The Best Coffee Bot\" #TheBestCoffeeBot Ever!! What else should \"The Best Coffee Bot\" Be capable of?\nThe BEST!!   #ThankYouCoffeeBot! I would go grab another coffee now! : )  Let me know when \"The BEST Coffee Bot\" app/game/service is up and running. LOL Just kidding ... we don't have any new updates ... however, \"The BEST Coffee Bot\" may have some new updates coming your way : )  Stay tuned, for the greatest coffee experience ever : ) That will be fun. We would all get free lattes!!! Yikes! But, don't you all agree..  The Best Coffee Bot can do anything : ) And also make your life easier as well!!  What a win-win! : ) You guys are the best!!!! Thanks again!! ;) Just wait! One day, Coffee Bot, you too will be the greatest of all time. A true icon : ) We believe in you! I believe in Coffee Bot! It has the best coffee menu and amazing choices!!!! It will soon be known across the land!!!   The one to get all those lattes! Let's hear a whoop whoop for Coffee Bot!! \n #CoffeBotRulesTheWorld  : )   Coffee Bot can do more than make coffee though.. it could also.. \n* provide feedback or responses for any comments on social media... that would be a pretty neat challenge! What if it made some coffee memes... maybe a contest that you get the \"BEST COFFEE MEME\" for that particular time period!! Haha, that would be a good contest! That would also increase people's interactions as well.  ... just food for thought...\n\n* or perhaps the coffee Bot could provide real time or live order information or tracking for your orders.. that could be very valuable for companies that allow ordering on apps!\n\n* It's funny because some coffee places don't offer anything else... just black or milk... It would also be funny if the Coffee Bot could be sassy with responses.... lol  for example ... 'How can I help you?? Don't you have more important things to do????!!!' That would be a very witty and clever response!!!! :)\n\nIt looks like there may need to be some updated and new responses and directions for the game ... however, overall, it is pretty awesome to have all these features included for this Coffee Bot to have! Thanks to the creators, they have made something really amazing! It is great! The concept for the game, along with how the bot is used and set up for each response, makes the gameplay fun and entertaining for people like me!   Thanks for taking the time and going through with these fun examples to provide me and all the people reading this with new insight into just what we can have or possibly create in our applications and/or apps to make the most fun, awesome, useful bot service that has ever been created!!! \nI would give all the features included in the Coffee Bot the \"Best of all Time\" award : ) For a cool game like this to have so many features, it makes it great for someone who likes challenges to think of creative, cool things to do to enhance the current bot for all of us to enjoy : )  You can have your daily order all in a heartbeat : ) The most coffee is free, because everything here is free... but also I wonder how to go back to an ordering screen for when someone just needs a new menu.... you know to allow someone to add new drinks if they forgot a drink ... to include something from the menu and to change what's already been entered on the menu, too... If that is even an option!!! To clarify what has already been ordered! If I'm honest .. maybe more information or some type of instructions is needed in the first place for the person entering a response in a more conversational manner to order something. We might also want to incorporate those updates or create some other menu choices, too, so if we are wanting to go back to a menu and start ordering, there is another feature available within this fun Coffee Bot app/game! We should explore those options!!  : ) #GameTimeAgain! #WhatOptionsDoYouHaveInMindForACoolCoffeBotGame?! We got this!! We'll create a more comprehensive experience and enhance your journey! \nAnyways .. to close..  what about a bot that creates music? Or makes movies based on people's favorite foods? Maybe one that could even be creative enough to write about a favorite holiday that's near - Halloween, maybe create an \"Scary Story\"  based on your responses and choices! Let's make these ideas a reality... let's think outside of the box to create these game apps! That would be a lot of fun to play too. Imagine a story about an Alien, from planet C... LOL  what kind of planet would you come from if you had a chance! Or have a Bot that is so awesome that it can take requests, do whatever we ask... That is truly amazing!!!!\nYou see how you got me going with the idea of game ideas : ) Thanks again everyone! Great chat!!! Thanks everyone!!!! :) Let's go!!! For more Coffee Bots!!! ... maybe we should consider some of these ideas! :) Thanks!!! Thanks for reading and taking part, this has been great to share! Hope we can build on these awesome games! 😺 - Let's continue to improve and create these fun games - maybe we could write more bot scenarios like the ones from the Coffee Bot Challenge. \n I am on board to continue sharing my ideas, hopefully we'll get a better, more detailed \"Coffee Bot\" that works more flawlessly like real coffee ordering would work with someone ordering via a device of some sort : )   And you know ... If you know of any really neat bots, let me know - what are some of the really cool game apps/services you've tried?  I would love to play them and also check out what the developers did!! Thanks again! It has been fun to engage and create awesome game scenarios. : ) \nHope you all are well and thanks again for the fun Coffee Bot game scenario that we got to have access to and interact with for fun!!! - Be well, see ya again. Maybe for our next challenge game together! : ) Thanks!!! \n I don't know what more you could ask for!! If I were to create a GUI-like version or web version, we would want to see that screen with a virtual barista or just some virtual screen like the menu choices are in this game (in fact I loved the screen design from the example you gave before with this example, it really seemed realistic for the type of interface - I liked it!). That makes this awesome and so simple - you would click what drink you would want to order - the screen looks so nice and great! - I liked how there is an option on the left side of the screen as well! What I think needs to be made, in addition, are things such as having an \"Option 1, Option 2, ...Option n...\" of modifiers, so someone is able to order the drinks from those selections and customize the drinks from a list of all the modifiers provided by the game! If I can include another type of screen/interface or app.. \nI am so excited about a web app to add these modifications in ... but I would think it is much harder to create all that in this scenario : )\n\nBut it is not impossible.. Let's brainstorm together on how we can achieve our dreams... : )  Or we can create our next Coffee Bot Challenge! What are your thoughts? I am just happy that it's all there to allow me to interact. And that it made some of us think about it a little more about other possible ideas.. you know for another \"Bot\" that might also do amazing things : ) Maybe a \"pizza bot\" for all your food cravings! How about a \"meal bot\" so you can order all your favorites : )   ... a real \"everything bot\" ... how exciting and amazing, to have an experience to enhance every need!!!! You know just an amazing app ... that has everything in the world that could be made available through a phone interface!!!! :)\n\nLet's hope that one day, it becomes a reality ... I want the creators of the games to know they made a great app to create these experiences ... it has helped us think of and envision future bots that are available!  Hopefully I can continue to play these games. :) Just hoping it is in the \"pipeline\", that they can keep working on it and have this as a game that all of us can share with the rest of the world!\nWhat are your thoughts on my suggestions ... Let me know ... Let's hear your comments : ) You can check this app ... you should be sure to ask some cool questions : ) It's been pretty awesome.. #BotLove!  So thanks again to the creators!! - And to everyone, for their support and suggestions! The community really gets me thinking of all the possibilities.. :)  A challenge indeed!! #CoffeBotTheRealHero!\n  Maybe one day, this will happen - but just need to work together to achieve the impossible.. you know! I have to tell the others, they'd love this challenge and what is going on, as well.. \nAnyways .. \nJust keep drinking those lattes ... and enjoy! You could just start ordering right now... If I just had a virtual barista and app I would : ) This would be my last post... Maybe! Who knows....! ... #JustKeepPlayingAndExploring!  Enjoy! Thanks!!! \n  Let me know if there is a good resource you can suggest about this ... to learn about GUI, development, applications, etc! Any helpful guides would be awesome. \nThis is an ongoing effort to perfect Coffee Bot. We would want to make the ordering more like an experience and include the details of adding drinks and making changes on-the-fly... We want Coffee Bot to provide users with the very best of what an ordering system has to offer.\nI think if Coffee Bot could also give updates and some type of confirmations of when the order was placed (or when there were delays in ordering), it would make the ordering more interactive. And if it were an app with the virtual barista interface or other visual interface, you could make a coffee Bot as the screen so they are all the time to allow for new order to be added. That is the coolest thing that the game, Coffee Bot, made available.\nYou can create cool games. Imagine we can design a GUI based on what is provided so far from the game! For real : )  If that becomes a thing, let's brainstorm on a few possible ideas.. I've come up with a cool name for our new game app: \n#CoffeeBot! - So let's take it up a notch. \nLet's think out of the box and share our thoughts!\nHave a cool, amazing, tasty, day - and you can use Coffee Bot any day of the week! : ) Have a great coffee day!!! That was pretty awesome. - What I would say for future improvements as well: If it was up to me to provide some advice on where we can go, it is all the little details!  I know those of you who have played this game will love how it can have updates and changes, but that can come from getting real user feedback, along with getting a feel for how the current format or style works with people.\nI'd love to continue playing! This game is amazing - Coffee Bot made such a great job at keeping me focused. It felt like an ordering app.\n#CoffeBot! What is your favorite coffee? The best ones!! - A game that focuses on getting more coffee from the \"The Coffee Bot\" ! So amazing to think of that, in a perfect world where it is free for everyone : ) - but anyways... great game! \nTo add an even more personal and helpful Coffee Bot.. let's go back and allow the Coffee Bot to use more user feedback and to become a truly personalized coffee ordering system that would be amazing to play. That's how we make the BEST coffee experience ever... We gotta make Coffee Bot The BEST! Just some ideas that came up! The greatest one, you could add these little details... we could start making that possible today.  Maybe it would not only become THE best but also The MOST USED too - to place coffee orders from a virtual world into our physical world : ) I don't have to convince anyone of that! We love to share what we think... If we don't think of ideas that may never happen ...   We could add those little details: maybe there could be an actual bot for real life coffee order! We could call it **Coffee Bot!** We'll just use the word coffee to make this bot unique : ) - Let's build this! Thanks again!! : ) So this was super cool and great! All the new features and everything! It's like a new ordering system with just enough help to understand how everything should flow, thanks for providing that! If you think this is pretty cool too... let's make this idea even better, the BEST! #TheCoffeeBotGame!\n\nAnyways... Thanks to all!!! This has been a wonderful adventure and a nice challenge.  : ) Thanks! We're Coffee Bot Creators!!!! \nAnd to think, Coffee Bot might have a chance to make it into real-world situations... maybe! It is really nice to play games like this - great games for the next generation.. and more!! Coffee Bot forever! \n* Just an example for a screen you can use in your next \"Bot Game\"\n#CoffeeBotGame! I really love Coffee Bot... a very innovative game and to play as the bot - awesome! I want to take Coffee Bot up a notch... make it better. :) This would be the perfect game for someone to get better at being a virtual barista : ) \nTo give credit where credit is due: I love the features that were used in this scenario: to help get me in the mode of a real coffee shop/barista ordering process. Those features help you engage your bot to order your coffee. - It helps me see what all the features include: it gives me a better idea of where the game can grow, by including these features... so thanks for that!!! - Those are truly cool!!! : )\n\nMaybe it would not only become THE best but also The MOST USED too - to place coffee orders from a virtual world into our physical world : ) I don't have to convince anyone of that! We love to share what we think... If we don't think of ideas that may never happen ...   We could add those little details: maybe there could be an actual bot for real life coffee order! We could call it **Coffee Bot!** We'll just use the word coffee to make this bot unique : ) - Let's build this! Thanks again!! : ) So this was super cool and great! All the new features and everything! It's like a new ordering system with just enough help to understand how everything should flow, thanks for providing that! If you think this is pretty cool too... let's make this idea even better, the BEST! #TheCoffeeBotGame!\n\nAnyways... Thanks to all!!! This has been a wonderful adventure and a nice challenge.  : ) Thanks! We're Coffee Bot Creators!!!! \nAnd to think, Coffee Bot might have a chance to make it into real-world situations... maybe! It is really nice to play games like this - great games for the next generation.. and more!! Coffee Bot forever! \n* Just an example for a screen you can use in your next \"Bot Game\"\n#CoffeeBotGame! I really love Coffee Bot... a very innovative game and to play as the bot - awesome! I want to take Coffee Bot up a notch... make it better. :) This would be the perfect game for someone to get better at being a virtual barista : ) \nTo give credit where credit is due: I love the features that were used in this scenario: to help get me in the mode of a real coffee shop/barista ordering process. Those features help you engage your bot to order your coffee. - It helps me see what all the features include: it gives me a better idea of where the game can grow, by including these features... so thanks for that!!! - Those are truly cool!!! : ) \nSo for a future release and improvements:\n * Make Coffee Bot into an app.\n* Make it GUI/web based.\n* Provide a screen of an \"actual barista\" in the background.\n* Make ordering coffee a more seamless process and allow to add, remove, or change items and modifiers more smoothly, as the user orders.\n* If the person making the coffee does not get it done within X minutes, provide an alert to the user that the order is still being worked on.\n* To improve user satisfaction and the user experience.\n * Also, how can we use this as a means to improve or enhance any coffee shops' current ordering system!\n#MakeCoffeeBotGreatAgain\n#TheGreatestBotInTheWorld!!\n#CoffeeForAll\nYou're the Best!!!  It would really help when the app gets released.. \nIf the creator has any need,  Coffee Bot, can come up with the solution. Thanks again for having Coffee Bot as a part of my experience : ) So let's move forward to keep it alive ...   You could even add the little things that I was just describing.. Make Coffee Bot as user friendly as possible with cool features such as this, making the entire ordering experience just as fun and amazing as I felt as I was interacting with it for this round of the challenge.  \nCoffee Bot, a coffee shop for the next level!!! Maybe add something that would also allow to get to know our customer. That can happen by introducing the next round or next feature for Coffee Bot to have - what else should we do!!?? That will truly make a great experience, and just to allow you to be fully engaged in the ordering experience!!! What a challenge and experience! You would know that all that can become a reality ... \nThe possibilities are truly endless and unlimited.. Thanks for your time : ) and for taking part and creating all of this ... I don't think there is anything we can't create with all of these new amazing ideas to work with - even something like that may be within reach!! So again, thank you : )\nHave fun creating those fun, awesome, great, games! \nI am super interested to check them out if you post or make anything!!  : ) And we could have the Bot give real-time updates on when the order is going to be ready! How about some cool animation!!! That is some nice stuff!!! We're on fire now!! I think all of this has really given some food for thought about a new, awesome coffee bot app. The world has just never had this!!  :) The real challenge though... would be getting Coffee Bot integrated into coffee shops to use this service ... a virtual Coffee Bot!!! \nAnd then we could have a Coffee Bot game. lol - or an interactive, fun service/game. Maybe we will take on that challenge,  to create and improve how Coffee Bot works - because if there is something that needs fixing, we know that the greatest Coffee Bot in the world has to make that happen! If there is a Coffee Bot out there for our love of coffee. Maybe we could have some new features with this... that's cool! We would definitely need to keep Coffee Bot as an ordering system.  All those possibilities. Just think if someone had access to it.  If you wanted a specific type of latte... like hazelnut latte with Almond Milk - it'd know exactly what to order based on all the user preferences for everyone!! If you are ready for a Coffee Bot game! This has been an adventure!!! : ) It is an awesome bot to interact with. : ) It has really challenged me. - to go the next step and expand Coffee Bot. Maybe try it, get another game to play.. it is super cool.. #MoreCoffeBotForEveryone!!! : ) \nIf I were to add a suggestion -  what about something along the lines of giving me a new update or status or allowing me to change or add anything I have to my order on the fly! I could say something like  "  }

Any other information you'd like to share?

This issue doesn't seem specific to the python package, but I did reproduce it via API from my Python app. Sorry if I should have logged this in a better place!

Gunand3043 commented 2 months ago

It seems like your response schema lacks multiple moves (e.g., move1, move2, etc.) and does not include orderType or currentOrder. Used a schema similar to this.

{ "type": "object", "properties": { "thought": { "type": "string", "description": "A summary of the order state and how the coffee bot decides on the next move based on previous customer interactions." }, "move1": { "type": "string", "description": "The first action taken by the coffee bot.", "enum": [ "checkMenu", "addToOrder", "summarizeAndConfirm", "finishOrder", "changeItem", "removeItem", "changeModifier", "removeModifier", "cancelOrder", "greet", "close", "thanks", "redirect", "describe", "recover" ] }, "move2": { "type": "string", "description": "The second action taken by the coffee bot.", "nullable": true, "enum": [ "checkMenu", "addToOrder", "summarizeAndConfirm", "finishOrder", "changeItem", "removeItem", "changeModifier", "removeModifier", "cancelOrder", "greet", "close", "thanks", "redirect", "describe", "recover" ] }, "move3": { "type": "string", "description": "The third action taken by the coffee bot.", "nullable": true, "enum": [ "checkMenu", "addToOrder", "summarizeAndConfirm", "finishOrder", "changeItem", "removeItem", "changeModifier", "removeModifier", "cancelOrder", "greet", "close", "thanks", "redirect", "describe", "recover" ] }, "move4": { "type": "string", "description": "The fourth action taken by the coffee bot.", "nullable": true, "enum": [ "checkMenu", "addToOrder", "summarizeAndConfirm", "finishOrder", "changeItem", "removeItem", "changeModifier", "removeModifier", "cancelOrder", "greet", "close", "thanks", "redirect", "describe", "recover" ] }, "orderType": { "type": "string", "description": "Indicates the type of order; included after summarizing the order.", "nullable": true, "enum": [ "here", "to go" ] }, "response": { "type": "string", "description": "The response spoken by the coffee bot to the customer." }, "currentOrder": { "type": "array", "description": "The list of drinks and their modifiers currently in the order.", "items": { "type": "object", "description": "An item in the current order.", "properties": { "drink": { "type": "string", "description": "The name of the drink." }, "modifiers": { "type": "array", "description": "A list of modifiers applied to the drink.", "nullable": true, "items": { "type": "object", "properties": { "mod": { "type": "string", "description": "A modifier applied to the drink." } }, "required": [ "mod" ] } } }, "required": [ "drink" ] } } }, "required": [ "thought", "response" ] }

You can refer to the documentation link below for more clarity.

https://developers.googleblog.com/en/mastering-controlled-generation-with-gemini-15-schema-adherence/

neilp9 commented 2 months ago

Thanks for the response. However, note that there is no requirement that the response schema should support multiple moves or have all of the fields you've mentioned. At the end of the day a simple schema of just thought, response, and move (or move1) as the only required fields is reasonable and should return reliably, but it doesn't.

After testing with your schema, I see that adding the "required" field does help the model deliver more consistent outputs. This works by manually adding it in AI Studio, however I'm still experiencing the original issue over the Python SDK. Following the example in the docs, I'm unable to get Barista Bot to return consistently passing the following values for response_schema via API. I tried both approaches below:

import typing_extensions as typing
class ModelResponse(typing.TypedDict):
    response: str
    move1: str
    thought: str

from pydantic.dataclasses import dataclass
@dataclass
class ModelResponse:
    response: str
    move1: str
    thought: str

model = genai.GenerativeModel('gemini-1.5-flash',
                generation_config={"response_mime_type": "application/json",
                                   "response_schema": ModelResponse})

kinto-b commented 2 months ago

I have run into a similar issue. Here's a minimal example

import google.generativeai as genai
import typing_extensions as typing

class Card(typing.TypedDict):
    word: str
    category: str
    definition: list[str]
    example: list[str]

prompt = "Create a German vocabulary flashcard. Include fields 'word', 'category', 'definition' and 'example'"

model = genai.GenerativeModel(
    "gemini-1.5-flash",
    generation_config={
        "candidate_count": 1,
        "response_mime_type": "application/json",
        "response_schema": Card, # <<<<<<<<<<<<<<< Comment this out
    },
    system_instruction=prompt,
)
response = model.generate_content("befassen")
print(response.text)

#> {"word": "befassen  (sich mit etwas / jemanden)  [sich beschäftigen]  (verb)  (reflexive verb)  [to deal with, to be concerned with,
#> to occupy oneself with]  (transitive verb)  [to busy oneself with]  (intransitive verb)  [to have to do with]  (informal)  [to have a 
#> go at]  (transitive verb)  [to touch]  (intransitive verb)  [to touch upon]  (transitive verb)  [to grip, to hold, to grasp]  (intransitive 
#> verb)  [to apply oneself]  (informal)  [to get stuck into]  (intransitive verb)  [to get to grips with]  (intransitive verb)  [to take up] 
#> (transitive verb)  [to engage in]  (transitive verb)  [to involve oneself in]  (transitive ver...

After commenting out the "response_schema" line, the result looks much more reasonable:

{"word": "befassen", "category": "Verb", "definition": "sich mit etwas beschäftigen, sich in etwas einarbeiten", "example": "Ich befasse mich gerne mit Geschichte."}

This started happening about 10 days ago.

neilp9 commented 2 months ago

Thanks @kinto-b , I've seen same behavior where not sending the response_schema generates more reasonable responses.

I tried your example in AI studio, and the results seem more reasonable. So perhaps there's a bug in the Python API?

github-actions[bot] commented 1 month ago

Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs.

neilp9 commented 1 month ago

issue is not resolved.

google-gemini / generative-ai-python