planning: Cortex Model Compatibility API

[imtuyethan]

Goal

• Cortex can generate a model compatibility prediction, based on user's hardware and model.yaml
• This should be an API that Jan can call (potentially as part of GET /models and GET /model/<model_id>)
• Likely linked to #1165
• Model compatibility should "compute" based on Active Hardware

Related Issues

• [ ] Update spec with architecture: https://github.com/janhq/cortex.cpp/discussions/1130
• [x] https://github.com/janhq/cortex.cpp/issues/1143
• [x] https://github.com/janhq/cortex.cpp/issues/1142
• [ ] https://github.com/janhq/cortex.cpp/issues/1140
• janhq/jan#1089

Original Post

Specs

https://www.notion.so/jan-ai/Hardware-Detection-and-Recommendations-b04bc3109c2846d58572415125e0a9a5?pvs=4

Key user stories

• Migrate Hardware Settings from Advanced Settings to its own settings page.
• Missing Hardware Dependencies → Ask users to download dependencies to turn on GPU acceleration.
• ...

  Design

  https://www.figma.com/design/DYfpMhf8qiSReKvYooBgDV/Jan-App-(3rd-version)?node-id=5115-60038&t=OgzCw09qXKxZj3DC-4

[dan-homebrew]

Note: This should be driven by Cortex team, with Jan UI as one of the task items.

I think this is part of a larger "Hardware Detection, Config and Recommendations"

• Cortex can detect what hardware the user has: CPU, GPU(s)
• Cortex can be instructed to use certain hardware (e.g. CPU-only, GPU, and which GPU)
• Cortex can run models with specific hardware (e.g. GPU 1)
• Cortex can assess hardware compatibility for a given model

[dan-homebrew]

This is also being discussed in janhq/jan#1089 - let's link both issues. We will need to scope this to something less ambigious

• e.g. Cortex can provide "recommendations" in GET /models, based on activated/detected hardware

[dan-homebrew]

Shifting to Sprint 21 to allow team to focus on Model Folder execution in Sprint 20