search

janhq / jan

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)
https://jan.ai/
GNU Affero General Public License v3.0
22.54k stars 1.3k forks source link

feat: Threads can override and inherit default params from Assistants, Models, and Engine #754

Closed dan-homebrew closed 9 months ago

dan-homebrew commented 10 months ago

Tasks

Spec

image

Overview

Wireframes

image image image image

Excalidraw: https://link.excalidraw.com/l/kFY0dI05mm/3WHim5p2OLu

dan-homebrew commented 10 months ago

We should ideally capture most if not all of the Model and Inference Engine settings, that Oobabooga, and etc avail.

Image

Image

dan-homebrew commented 10 months ago

See LiteLLM's approach to prefix, suffix and roles and prompt templates

https://github.com/abetlen/llama-cpp-python/issues/717#issuecomment-1722093969

Image

dan-homebrew commented 10 months ago

One question is also whether there should be a "Hardware Settings", e.g. for Apple Silicon users? Is this also present for other types of Hardware - e.g. Intel, etc.

https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_increase_vram_allocation_with_sudo_sysctl/

tikikun commented 10 months ago

@dan-jan for current release there will be no different in settings for different hardwares, mostly already automatically handled at build time

tikikun commented 10 months ago

important asset reference for prompt template: https://github.com/mounta11n/plusplus-caMalL/blob/4fa32ad0e3b61a6e13cdb6261723b6fd608a0fad/examples/server/public/promptFormats.js#L15

tikikun commented 10 months ago

Q:Where are these prompt template coming from? Most of community projects now using TheBloke metadata as referrence, example.

https://huggingface.co/TheBloke/LLaMA2-13B-Psyfighter2-AWQ#prompt-template-alpaca-tiefighter

image
tikikun commented 10 months ago

We will have 2 main modes: LOCAL and REMOTE

LOCAL

image

REMOTE

image
tikikun commented 10 months ago

Link to excalid draw: https://excalidraw.com/#json=kOBPg9OoLTCLAm3JO7FHn,qV29wMh7fLvGkFXf5HRYNA

0xSage commented 10 months ago

Asks:

  1. Can we the settings properties in swagger/json schema (with validation, etc).
  2. A list of prompt templates (so users can have a dropdown selector)

schema fields

id: 
type: 
required: boolean
data_validation: {
  min: 
  max:
}
default:
information: // (i) that tells users what this field actually does
0xSage commented 10 months ago

Discussion updates, mostly as a reminder/reference for @imtuyethan

Based on: https://app.excalidraw.com/s/kFY0dI05mm/3WHim5p2OLu

  1. Assistant Section: Add back the model selector

    • Users should be able to choose a model to chat with when using Jan assistant.
    • Remember: users can only change models at the beginning of a thread, not after thread has started
    • So we need to also show a change-model-disabled-state in designs.
    • Remember: the model determines the variables in the next sections: inference parameters, model params engine params content may change depending on the model chosen.

  2. Users can change Assistant Instructions aka the system message

    • Users can edit assistant instructions after conversation starts.
    • We briefly discussed sending a new “system message updated... etc.” In the MainPanel. Ashley will play around with a potential design for it.
    • The system message may show up in MainPane differently, not as a regular message sent from the user, but as a little box/styled differently?
    • Again, this is a nice to have. It's possible we don't show anything related to it in the MainView at all

  3. We shouldn't auto save when users edit each field. We should make users click a global save button, which saves all sections at once.

    • Reason: reduce fs writes to thread.json
    • See (4)

  4. Users must restart model after editting model_params or engine_parameters

    • If users edit model_parameters & engine_parameters, then they have to save & restart model, a two step process. @imtuyethan will play around with how the designs communicate this.
    • These 2 sections should say “requires model restart” somewhere...

  5. Alan is working on parameter fields, like mins/max/etc.

  6. Model Parameters section UI

  7. Actually play around with having a reset button for each sections, at the section level (not global)

  8. @Rex make sure every model has a prompt template filled in

  9. Alan TODO later: low fidelity mock on how “model.json settings is different than thread.model.json settings”

  10. Remote model side panel - we didn't discuss this as much. Punt till later

0xSage commented 10 months ago

@imtuyethan

A note on the menu items, I saw that we currently open up model.json assistant.json when users click "Reveal in Finder/JSON" in all the sections.

It should be

Image

namchuai commented 9 months ago

@imtuyethan , please help to verify this. cc @louis-jan