search

microsoft / teams-ai

SDK focused on building AI based applications and extensions for Microsoft Teams and other Bot Framework channels
MIT License
396 stars 170 forks source link

[Custom Engine Copilot] Teams AI Bot returns an error "Reached max model response repair attempts" #1899

Open PjPraveenkumar opened 1 month ago

PjPraveenkumar commented 1 month ago

Please be sure to check the Discussions Q&A section before filing a new question.

Question

Issue 1: The bot consistently provides accurate responses but also includes an error message, as shown below: image

Issue 2: The AI bot’s response appears at the bottom of the "actual response" instead of at the top. image

Issue 3: If no action handler is registered for an unknown query, the Teams AI returns the following error. We tried using the unknown action handler, but this did not resolve the issue. how can we handled the unknown query in this case if no actions were registered ? image

Code snippets

What you have tried already

For issue1: currently partner is returning the response to the Teams AI bot in HTML format, though it expects the response in proper JSON format. We updated the model properties to use JSON format and adjusted the skprompt.txt file to request JSON responses explicitly, but this didn’t resolve the issue. We also attempted to immediately return a 'STOP' command upon receiving the actual response from the bot to cut out the error message, but this is not a complete solution.

For issue 2: Implementing the STOP command right after the actual Bot response eliminated the bottom summary message returned by the AI, but ideally partner wants the summary message at first as we see in M365 copilot chat.

For issue 3: We tried using the unknown action handler, but this did not resolve the issue.

Note: please use this template to file your questions regarding implementation. Once your question is answered, our team will move this question to Q&A in Discussions for findability.

corinagum commented 1 month ago

Hi @PjPraveenkumar, I have some follow up questions:

  1. What AI system/model are you using? Are these the latest versions available?
  2. What type of augmentation are you using? There's currently default/none, monologue, and sequence augmentations. You can check your config.json for this. if augmentations object is missing or empty, that is default augmentation.
  3. Have you enabled logging in your AI module? When you enable logging and check the console after these responses are received, what does the response JSON look like? Is it malformed in any way?
  4. What is your max_repair_attempts number set to? If you try bumping this number, how do the response attempts change, if at all?
  5. For issue 1, is the error message being sent at the same time as the response message, or different time? In the logger, what order do the error message and correct response come in? Seeing your logging for this would be very helpful, because at first glance it looks like multiple calls are being sent to the API. What does the prompt and handler code look like for this scenario?
  6. For issue 2, same thing, could you share the logging order and response JSON's for this prompt?
  7. Without seeing your code, it looks like you are correctly addressing the issue for 3, which makes me suspect there is a deeper issue that is causing all three problems listed. Could you share the handler code for UnknownResponse?
PjPraveenkumar commented 1 month ago

hello @corinagum please find the answers below,

  1. What AI system/model are you using? Are these the latest versions available? ANS: Azure OpenAI service – Model: gpt4-32k

  2. What type of augmentation are you using? There's currently default/none, monologue, and sequence augmentations. You can check your config.json for this. if augmentations object is missing or empty, that is default augmentation. ANS: partner was using “sequence”. also they tried utilizing the “monologue” but didn’t work.

  3. Have you enabled logging in your AI module? When you enable logging and check the console after these responses are received, what does the response JSON look like? Is it malformed in any way? ANS:
    partner have enabled logging but didn’t see the complete response in the debug logs (I've attached the logs we currently have). Could you provide guidance on how to capture the JSON response in the logger to determine if it's malformed?

  4. What is your max_repair_attempts number set to? If you try bumping this number, how do the response attempts change, if at all? ANS: The partner has currently set max_repair_attempts to 5 and observed that it can now perform successful repair attempts. They did not encounter the error message along with the actual response (e.g., issue 1).

  5. For issue 1, is the error message being sent at the same time as the response message, or different time? In the logger, what order do the error message and correct response come in? Seeing your logging for this would be very helpful, because at first glance it looks like multiple calls are being sent to the API. What does the prompt and handler code look like for this scenario? ANS: - it was the same time as the actual response message, I have attached the error log test file for reference(error 1).

  6. For issue 2, same thing, could you share the logging order and response JSON's for this prompt? ANS: - I have attached the logging order in the text file(error 2).

  7. Without seeing your code, it looks like you are correctly addressing the issue for 3, which makes me suspect there is a deeper issue that is causing all three problems listed. Could you share the handler code for UnknownResponse?

ANS: The partner used the default actions (unknown action handler), but it did not get triggered when no action handler was registered for an unknown query. please see the snap below. image Error 1.txt Error 2.txt Error 3.txt

corinagum commented 1 month ago

@PjPraveenkumar Thank you, could you please provide the ai module's console logging? When creating the app, your aiOptions should include logger factory: https://github.com/microsoft/teams-ai/blob/main/dotnet/packages/Microsoft.TeamsAI/Microsoft.TeamsAI/AI/Models/OpenAIModel.cs#L49

The stack tracing you have provided is helpful to a point, but the logger info would be most helpful.

corinagum commented 1 month ago

@PjPraveenkumar looking at error 3.txt, it appears that there's a login error to bot framework. Please check authentication.

PjPraveenkumar commented 3 weeks ago

Hi @corinagum The login error has been resolved, but the partner is still encountering the same issue. Could you please confirm how the partner should handle unknown queries, given that no action handler has been registered? Additionally, is there a C# sample reference available that demonstrates how to address this situation? partner tried utilizing the default unknow action handler but it doesn't help. image

And the AI bot's response is appearing at the bottom of the "actual response" instead of at the top. Could you provide some suggestions on how to address this issue? image

PjPraveenkumar commented 2 weeks ago

Hi @corinagum could you please let us know if there is any update on the above issue ?

corinagum commented 2 weeks ago

@PjPraveenkumar at latest update, I had asked you for the log entries from turning on logging in the AI module for the remaining issues. Until I get those, it will be difficult for me to support you further.