janhq / jan

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)
https://jan.ai/
GNU Affero General Public License v3.0
22.06k stars 1.27k forks source link

feat: Jan supports most llama.cpp params #3508

Open imtuyethan opened 2 weeks ago

imtuyethan commented 2 weeks ago

Goal

Tasklist

Cortex

Jan

Related

Original Post

- [X] I have searched the existing issues

### Is your feature request related to a problem? Please describe it

- llama.cpp Settings (e.g. attention) should be consistent across llama.cpp, Cortex and Jan
- From an Eng perspective, we should ensure llama.cpp settings get bubbled up to Cortex and Jan

### Describe the solution

- [ ] Identify all relevant model settings that need to be synced
- [ ] Design a common format for representing these settings across all projects
- [ ] Jan Model Settings should follow common format
- [ ] Cortex should allow user to pass inference-time and runtime parameters
- [ ] Process for llama.cpp updates (who should this be driven by?)

### Teachability, documentation, adoption, migration strategy

-

### What is the motivation / use case for changing the behavior?
0xSage commented 1 week ago

Related #3140