search

defenseunicorns / leapfrogai

Production-ready Generative AI for local, cloud native, airgap, and edge deployments.
https://leapfrog.ai
Apache License 2.0
260 stars 29 forks source link

bug(ui): assistant chat model doesn't update when backend is changed #847

Open gphorvath opened 4 months ago

gphorvath commented 4 months ago

Environment

Device and OS: Desktop Windows 11 Pro / WSL2 (Ubuntu 24.04) App/package versions: This commit from main, dev build: ea159500c063fc3122bde4aaa17e845eceaa30ed Kubernetes distro being used: k3d-slim:0.24.1 w/ GPU

Steps to reproduce

  1. Deploy UI with llama-cpp-python as the default model backend
  2. Create and use an assistant
  3. Redeploy UI with vllm as the default model backend
  4. Attempt to use the assistant (doesn't work)
  5. Create and using a new assistant (does work)
  6. Attempt to use first assistant (still doesn't work)

Expected result

Expect that the assistant will either use the default model, or allow it to be configured by the user.

Actual Result

Assistant continues using the old model backend and can't be convinced to use new model backend.

Visual Proof (screenshots, videos, text, etc)

image

Severity/Priority

Low/Low

Additional Context

Add any other context or screenshots about the technical debt here.

andrewrisse commented 3 months ago

@barronstone @gregclark-defenseunicorns This story needs some more user exploration for how we want to handle this.

Should the user be able to assign specific models to assistants? Or should the assistant dynamically swap the model based on some preference?