planning: Provider Abstraction

dan-homebrew commented 4 days ago

Goal

We should focus on this, and then later do Remote API Extension and Local

Tasklist

[ ] #3690
[ ] #2998

dan-homebrew commented 4 days ago

Goal: Clear Eng Spec for Providers

Scope

"Provider" Scope
- Remote (Groq, NIM, etc)
- Local = Hardware + APIs + Model Management (Ollama, Cortex)
- It is possible that we don't need a differentiation between Remote and Local
- Choosing a better name, vs. "OAIEngine")
Provider Interface + Abstraction
- Providers registers certain things (e.g. UI, Models), can be called by other extensions
- Registers Settings Page
- Registers Models Category + List
Each Provider Extension should be a separate repo?
- I would like this -> add others to help maintain

Jan Providers

Local Provider

Currently, the local extension still has to manage processes itself, which involves utilizing third-party frameworks such as Node.js (child_process) for building functions.

What if we build Jan on mobile we have to cover extensions as well. It would be better to move these parts to Core module and frontend will just need to use it’s API.

Local Provider will need to execute a command to run its program. Therefore, the command and arguments will be defined, while the rest will be delegated to the super class.

Lifecycle:

A Local Provider is intended to run engines as an API Server (potentially using HTTP, socket, or gRPC).
Local Provider executes a command through CoreAPI (reducing the main process implementation from extensions, easy to port to other platforms such as mobile)
Main Process core module will run a watchdog and maintain the process
Since then, the app has been able to make requests and proxy through the Local Provider extension.
App terminates -> watchdog terminates the process.

Examples

class CortexProvider extends LocalProvider {
  async onLoad() {
     // The Run is implemented from the core module
     // then the spawn process will be maintained by the watchdog
     this.run("cortex", [ "start", "--port", "39291" ], { cwd: "./", env: { } })
  } 

  async loadModel() {
    // Can be a http request, socket or grpc
    this.post("/v1/model/start", { mode: "llama3.2" })
  }
}

https://drive.google.com/file/d/1lITgfqviqA5b0-etSGtU5wI8BS7_TXza/view?usp=sharing

Remove Provider

The same as #3505
Remote extensions should work with autopopulating models, e.g. /models list.
We could not build hundreds model.json files manually.
The current extension framework is actually designed to handle this, it's just an implementation issue from extensions, which can be improved.
There was a hacky UI implementation where we pre-populated models, then disabled all of them until the API key was set. That should be a part of the extension, not the Jan app.

Extension builder still ships default available models.

// Before
override async onLoad(): Promise<void> {
  super.onLoad()
  // Register Settings (API Key, Endpoints)
  this.registerSettings(SETTINGS)

  // Pre-populate models - persist model.json files
  // MODELS are model.json files that come with the extension.
  this.registerModels(MODELS)
}

// After
override async onLoad(): Promise<void> {
  super.onLoad()
  // Register Settings (API Key, Endpoints)
  this.registerSettings(SETTINGS)

  // Fetch models from provider models endpoint - just a simple fetch
  // Default to `/models`
  get('/models')
    .then((models) => {
        // Model builder will construct model template (aka preset)
    // This operation builds Model DTOs that works with the app.
    this.registerModels(this.modelBuilder.build(models))
    })
}

Remote Provider Extension

Draw.io

https://drive.google.com/file/d/1pl9WjCzKl519keva85aHqUhx2u0onVf4/view?usp=sharing

Supported parameters?
- Each provider works with different parameters, but they all share the same basic function with the current ones defined.
- We've already supported transformPayload and transformResponse to adapt to these cases.
- So users still see parameters consistent from model to model, but the magic happens behind the scenes, where the transformations are simplified under the hood.
```
/**
```
- transformPayload Example
- Tranform the payload before sending it to the inference endpoint.
- The new preview models such as o1-mini and o1-preview replaced max_tokens by max_completion_tokens parameter.
- Others do not. */ transformPayload = (payload: OpenAIPayloadType): OpenAIPayloadType => { // Transform the payload for preview models if (this.previewModels.includes(payload.model)) { const { max_tokens, ...params } = payload return { ...params, max_completion_tokens: max_tokens } } // Pass through for officialw models return payload }
Decoration?
- We've currently hard-coded many provider metadata from Jan, which could cause issues with future installed extensions.
- The decoration should be done from the Extension Manifest (package.json).
- https://code.visualstudio.com/api/references/extension-manifest
```
{
"name": "openai-extension",
"displayName": "OpenAI Extension Provider",
"icon": "https://openai.com/logo.png"
}
```
Just remove the hacky parts
- Model Dropdown: It checks if the engine is nitro or others, filtering for local versus cloud sections. New local engines will be treated as remote engines (e.g. cortex.cpp). -> Filter by Extension type (class name or type, e.g. LocalOAIEngine vs RemoteOAIEngine).
- All models from the cloud provider are disabled by default if no API key is set. What if I use a self-hosted endpoint without API key restrictions? Models available or not should be determined from the extensions, when there are no credentials to meet the requirements, it will result in an empty section, indicating no available models. When users input the API-Key from extension settings page, it will fetch model list automatically and cache. Users can also refresh the models list from there (should not fetch so many times, we are building a local-first application)
- Application settings can be a bit confusing, with Model Providers and Core Extensions listed separately. Where do other extensions fit in?

Extension settings do not have a community or "others" section

Provider Interface and abstraction

Providers are scoped at engine operations, such as running engines, loading models...
- registerModels(models)
- run(commands, arguments, options)
- loadModel(model)
- unloadModel(model)
Core functions can be extended through extensions, not confined by providers, such as Hardware and UI.
- systemStatus()
- registerSettings()
- registerRibbon()
- registerView()

Registered models will be stored in an in-memory store, accessible from other extensions(ModelManager.instance().models). The same as settings. App and extensions can perform chat/completions requests with just model name, which means the registered model should be unique across extensions.

The core module also exposes extensive APIs, such as systemStatus so other extensions can access, there should be just one implementation of the logic supplied by extensions. Otherwise, it will merely be utilized within the extension, first come, first served.

The UI of the model should be aligned with the model object, minimize decorations (e.g. model icon), and avoid introducing various types of model DTOs.

Each Provider Extension should be a separate repo?

Extensions installation is a straightforward process that requires minimal effort.

There is no official way to install extensions from a GitHub repository URL. Users typically don't know how to package and install software from sources.
There should be a shortcut from the settings page that allows users to input the URL, pop up the extension repository details, and then install from there. It would be helpful to provide a list of community extensions, allowing users to easily find the right extension for their specific use case without having to search.

dan-homebrew commented 2 days ago

@louis-jan We can start working on this refactor, and make adjustments on the edges. Thank you for the clear spec!

janhq / jan