louis-jan
commented
2 months ago Scope of changes
- Nitro Inference Extension
- Model Extension
- Monitoring Extension
Nitro inference extension
Current implementation
-
Register Models (pre-populate model.json files) Any extensions register models on load will pre-populate
model.jsonunder/models/[model-id]/model.jsonsequenceDiagram participant ModelExtension participant BaseExtension participant FileSystem ModelExtension->>BaseExtension: Register Models BaseExtension->>BaseExtension: Pre-populate Data BaseExtension->>FileSystem: Write to /models - Load Model:
- Set additional .dll/.so PATH (for engine loading)
- Hardware Information (to decide engine binary)
- Run nitro server
- Parse prompt template
- Load a GGUF model with its file path and model settings (passed from App)
sequenceDiagram
participant App
participant NitroInferenceExtension
participant NitroServer
App->>NitroInferenceExtension: loadModel
NitroInferenceExtension->>NitroInferenceExtension: killProcess
NitroInferenceExtension->>NitroInferenceExtension: fetch hardware information
NitroInferenceExtension->>child_process: spawn Nitro process
NitroInferenceExtension->>NitroServer: wait for server healthy
NitroInferenceExtension->>NitroInferenceExtension: parsePromptTemplate
NitroInferenceExtension->>NitroServer: send loadModel request
NitroInferenceExtension->>NitroServer: wait for model loaded- Inference (inheritance - OAIEngine.ts) Any extensions inheriting from the Base OAI Engine class will forward requests to their respective inference endpoints.
sequenceDiagram
participant App
participant NitroInferenceExtension
participant NitroServer
App->>NitroInferenceExtension: inference
NitroInferenceExtension->>NitroInferenceExtension: transform payload
NitroInferenceExtension->>NitroServer: chat/completions
Possible Changes
| Current ❌ | Upcoming ✅ |
|---|---|
| Run Nitro server on model load | Run cortex.cpp daemon service on start |
| Kill nitro process on pre-model-load and pre-app-exit | Keep cortex-cpp alive, daemon process, stop on exit |
| Heavy hardware detection & prompt processing | Just send a request |
| So many requests (check port, check health, model load status) | One request to do the whole thing |
| Mixing of model management and inference - Multiple responsibilities | Single responsibility |
Model extension
Current implementation
- Download Model (ModelFile as payload)
- Delete Model (ModelFile as payload)
- Get Models (Scan through models folder and return ModelFile[])
- Import Model (Generate ModelFile and download)
- Fetch HF Repo Data (for HF model import selection)
App retrieves pre-populated models:
sequenceDiagram
App ->> ModelExtension: get available models
ModelExtension ->> FS: read /models
FS --> ModelExtension : ModelFileApp downloads a model:
sequenceDiagram
App ->> ModelExtension: downloads
ModelExtension ->> Networking : request
Networking ->> FileSystem : filestream
Networking --> ModelExtension : progress
App imports a model
sequenceDiagram
App ->> ModelExtension: downloads
ModelExtension ->> Networking : request
ModelExtension ->> model.json :generate
Networking ->> FileSystem : filestream
Networking --> ModelExtension : progressApp deletes a model
graph LR
App --> |remove| Model_Extension
Model_Extension --> |FS unlink| /models/model/__files__Possible Changes
| Current ❌ | Upcoming ✅ |
|---|---|
| Implementation - Depends on FS | Abstraction - API Forwarding |
| List Available Models: Scan through Model Folder | GET /models |
| Delete: Unlink FS | DELETE /models |
| Download: Download | POST & Progress /models/pulls |
| Broken Model Import - Using default model.json | cortex.cpp handles the model metadata |
| Model prediction depends on model size & available RAM/VRAM only | cortex.cpp predicts base on hardware and model.yaml |
System Monitoring extension
Current implementation
- Get GPU Settings
- Get System Information
App get resources information
graph LR
App --> |getResourcesInfo| Model_Extension
Model_Extension --> |fetch| node-os-utils
Model_Extension --> |getCurrentLoad| nvidia-smiPossible Changes
| Current ❌ | Upcoming ✅ |
|---|---|
| Implementation - Depends on FS & CMD | Abstraction - API Forwarding |
| Execute CMD | GET - Hardware Information Endpoint |
Overview
| Current ❌ |
|---|
| Upcoming ✅ |
Assumption
- cortex.cpp bundles multiple
engines(different CPU instructions and CUDA versions) - cortex.cpp support
/modelsAPIs- GET: /models (available, active status, compatibility prediction)
- POST: /models/pull (& progress?)
- DELETE: /models
- cortex.cpp support
/hardware-informationAPI
Challenges of moving Nitro to cortex.cpp
- Different Data (Folder & File) structures
- Backward / Forward compatibility
The migration
- How to seamlessly move from Nitro to cortex.cpp, where:
- cortex.cpp works with new Data Folder structure
- cortex.cpp works with model.yaml
- cortex.cpp works with
models.list
- How to maintain the data folder when users switch back to older versions?
- Older versions rely on
model-extension, which searches for amodel.jsonfile within the Data Folder. - Newer versions rely on
cortex-extension, which searches for amodel.yamlfile within the Data Folder.
- Older versions rely on
Let's think about a couple of our principles.
- We don't remove or manipulate user data.
- Rollback should always work.
-
Minimal migration
What are some of the main concerns here?
- Can we use
model.jsonandmodel.yamlside by side?- We should. Since the model folder can contain anything, from README.md, .gitignore, GGUF, model.yaml to model.json.
- Older versions will still function with legacy
model.jsonfiles. - Newer versions will work with the latest model.yaml files.
- How to sync between those two?
- It's hard to sync between those two, since different structures could break the app.
- We just try to migrate once when there is no models.list available. This is a good flag for migration triggering.
- After migrating, each app version works independently with its own model file format.
- How about model pre-population? In other words, Model Hub.
- Model pre-population is an anti-pattern. Pre-populated models do not work with versioning or create unwanted data that confuse users. How about our Model Hub list thousands of models?
- We implemented model import, which replaces the need for a model file. Users can just import with the HF repo ID. Users do not have any reason to duplicate or edit a pre-populated model.json.
- Model listing can be done from the extension.
- In short, in the next version, we don't pre-populate unwanted files to the Data Folder. Only when users decide to download.
- Users deleting a model means deleting the persisted model.yaml & model files.
- How do other extensions work with their
models? E.g., OpenAI- Remote models can be populated during the build, not persisted.
registerModelsnow persists in-memory model DTO. - We don't pre-populate remote models, which is not necessary. Users are better setting them from Extension Settings. It's more or less an Extension configuration, not Model population.
- Remote models can be populated during the build, not persisted.
- Migration complexity and UX
- We don't convert model.json to model.yaml. Instead, import with symlink. It could be faster and avoid new logic added from Jan, which is redundant. Lightweight migration with less risk. Maintain the Model ID is key; otherwise, all threads break.
- We don't move any files, which could drag the migration process long. E.g., GGUF
- How about new/manual adding GGUFs? The model symlink feature is always there for that.
- There are bad migration experiences in the past that we can avoid such as:
- Migrate all pre-populated models
- Heavy file movement drags the duration long
- Migrate everything at once
- Now we just migrate downloaded models:
- Import downloaded models only as symlinks (no file movement)
- Don't update the ID, which will kill us on data inconsistency
- Another thought: Do we really need to wait for model.yaml creation during migration?
- cortex.cpp can work with the
models.listto provide available models? - model.yaml generation is an asynchronous operation so:
- It generates
model.yamlas soon as user try to get or load. - It generates
model.yamlas soon as user try to import. - Don't block the client GUI;
model listcan be done with just themodels.listcontents. Any further operations with a certain model can generate amodel.yamllater. - Client will prioritize the active thread's model then others to not blocking users working threads.
- If something goes wrong, the GGUF file will still be there and can be generated later on other operations. The model.yaml file is not strictly required to be available, but just the cache of model file metadata?
- It generates
- cortex.cpp can work with the
- Better cache mechanism
- Model list and detail have worked with the File System before, and now they're sending an API request to cortex.cpp.
- To prevent slow loading, the client should cache accordingly on the frontend.
Summary
In short, the entire migration process is just to create symlinks to downloaded models from
models.list. Nomodel.yamlor folder manipulation involved. It should be done instantly?
Migrate indicator: models.list exist.
Don't pre-populate models. Remote Extensions work with their own settings instead of pre-populated models. Cortex Extension registers in-memory available to pull models (templates).
cortex.cpp is a daemon process that should be started alongside the app.
| Jan migration from 0.5.x to 0.6.0 |
|---|
0xSage
dan-homebrew
Goal
nitro-extension, andcortex-extension?)Tasklist