dimagi / open-chat-studio

A web based platform for building Chatbots backed by Large Language Models
BSD 3-Clause "New" or "Revised" License
14 stars 7 forks source link

Removing models does not alert experiments #568

Open bderenzi opened 2 months ago

bderenzi commented 2 months ago

Right now the model used is determined by the llm string and the llm_provider_id. If we change the available llm models (e.g., because a provider is deprecating old models), the experiment is never alerted / there is no check.

Possible solutions:

That solution is a little annoying for the user unless in the same edit-provider UI we allow them to select a replacement. For example, imagine the case where we're removing llama-3 and adding llama-3.1. The steps would be:

  1. Edit the provider to remove llama-3 and add llama-3.1
  2. Get an error that projects X, Y, Z are all reliant on llama-3 so we can't remove them yet
  3. Change the edit to just add llama-3.1
  4. Go manually update all projects relying on the old model
  5. Return to the edit provider screen to remove llama-3

An alternative:

  1. Edit the provider to remove llama-3 and add llama-3.1
  2. Get an error that projects X, Y, Z are all reliant on llama-3 so we can't remove them yet. Ask for a replacement model that will be applied to all experiments that are affected
  3. Confirm replacement and be done.
SmittieC commented 2 months ago

Option 2 is deffs better UX. +1 for that