search

microsoft / teams-ai

SDK focused on building AI based applications and extensions for Microsoft Teams and other Bot Framework channels
MIT License
391 stars 169 forks source link

[Bug]: gpt-4o and gpt-4o-mini models not working for assistants #1861

Open stuart-pebble opened 1 month ago

stuart-pebble commented 1 month ago

Language

Javascript/Typescript

Version

latest

Description

When using OpenAI Assistants, only some models work, not all of them. None of the gpt4o variants work. Its worth noting that gpt-4-1106-preview and gpt-4-turbo models do work.

Reproduction Steps

1. Build an MS Teams Assistant API bot using the defaults 
2. Modify the OpenAI Assistant to use the preferred model
3.
...
corinagum commented 1 month ago

Hi @stuart-pebble, which sample are you using? Order bot? or Math bot? What errors are you getting?

stuart-pebble commented 1 month ago

I'm not using a sample, I'm using the MS Teams Toolkit to create one from scratch using:

Create New App image

Custom Copilot: image

AI Agent: image

Build with Assistants API (Preview) image

JavaScript image

My objective is a simple RAG use case where users can ask questions with responses based on RAG from an OpenAI configured vector store. I can work around the gpt-4o-mini issue and select the gpt-4-turbo model in the short term but its not optimal.

FYI: I'm also having the same issues as https://github.com/microsoft/teams-ai/issues/1585 where, using the defaults, the OpenAI RAG functions are ignored. I'm currently exploring options to either update the node module or redeploy using the orderbot example.

corinagum commented 1 month ago

@stuart-pebble what errors are you getting? What is your sample code? We will need information in order to determine the bug.

stuart-pebble commented 1 month ago

If I configure the Assistant in OpenAI with gpt-4o or gpt-4o-mini all I see in the MS Teams chabot is " There was an error". If I change to one of the other models it just works.

lariar commented 4 weeks ago

Came here to confirm also seeing this problem. Took 3 days to figure out my own chatbot would work by reverting it back to GPT-4-turbo on the OpenAI side.

yikwen commented 3 weeks ago

I faced this issue as well where older GPT models worked fine and things started breaking when changed to GPT-4o. Inspecting the completions coming from GPT-4o, I found that the issue was with GPT-4o not following instructions in creating the plan using SAY and DO commands (I'm using sequence augmentation btw).

I added the following sentences to skprompt.txt and it performed better subsequently.

  • Provide your response in the form of a JSON object following the format provided at the end of these instructions
  • If the user's request matches an action you can perform, add that action to the JSON plan as a DO command. For anything you want to ask or tell the user, add the text to the JSON plan as a SAY command.
Stevenic commented 2 weeks ago

Thanks for the hint @yikwen. We're going to look into using the new structured outputs support for plan generation which should help resolve the issue.

Stevenic commented 2 weeks ago

@stuart-pebble I think the issue you're hitting is something different so I'll try to reproduce it

Benjiiim commented 5 days ago

We're going to look into using the new structured outputs support for plan generation which should help resolve the issue.

The perspective to have Teams AI Library using structured outputs and GPT-4o-mini is really exciting! I can't wait to see the impact it will have, especially on cost-effectiveness. Is this something that we can expect to get our hands on soon?