search

dimagi / open-chat-studio

A web based platform for building Chatbots backed by Large Language Models
BSD 3-Clause "New" or "Revised" License
13 stars 7 forks source link

Multi-Prompt Bot configuration not working with Assistants API #458

Open dwarren214 opened 2 weeks ago

dwarren214 commented 2 weeks ago

Description: The new Multi-Prompt Bot configuration in Open Chat Studio (OCS) is not functioning correctly with the OpenAI Assistants API.

Issue:

Router Prompt in Assistants API: Instead of routing messages to the appropriate subordinate prompts, the Router Prompt appears to respond to all messages itself.

Tests:

Tests using the Assistants API in OpenAI's Playground are successful, with the Router bot returning just the name of the subordinate bot that should be responding to the user's message. Examples:

Assistants API (Not Working)

Concept_Note_Router: Link Concept_Note_Composition_Assistant: Link Concept_Note_Brainstorm_Assistant: Link

Chat Completions API (Working)

Concept_Note_Router2: Link Concept_Note_Brainstorm_Assistant2: Link Concept_Note_Composition_Assistant2: Link

snopoke commented 2 weeks ago

One immediate note is that the prompt for the 'router2' doesn't use consistent naming from the routes. In some cases the 2 is appended and in other cases it isn't. That could confuse the model

dwarren214 commented 2 weeks ago

Thanks for pointing that out. That allowed me to realize that I have misidentified the provided examples. I will edit and correctly label them now. The Router 2 example is the chat completions API case, which is working--despite the naming convention errors. Sorry for the mix up.

SmittieC commented 2 weeks ago

I have an untested theory that a "normal" bot for the router will work better in all cases. So we still stick to all-or-no assistant bots for the subordinate ones, but we keep the router a normal bot, since all it should do is route. This should not have an impact on the conversation history, since we only record the user-bot interactions, not the routing itself :thinking: