bug: wrong max context with Command-R

[Kep0a]

Jan version

0.5.9

Describe the Bug

Locally imported Command-R 08-2024 GGUF has an incorrect maximum context length of 8192.

Command-R 08-2024 has a maximum context of 128k.

Potential solution: locally imported model inference settings should all be able to be manually set

Steps to Reproduce

1. Sym-link c4ai-command-r-08-2024-Q5_K_M.gguf
2. context length cannot be extended past 8192

Screenshots / Logs

What is your OS?

• [X] MacOS
• [ ] Windows
• [ ] Linux

[imtuyethan]

@Kep0a For self-imported models, you'll need to configure the model settings yourself since Jan won't be able to automatically detect the optimal settings.

How to manually configure model parameters

1. After importing your model, go to your Jan Data Folder (you can open it from Settings > Advanced Settings > Jan Data Folder)
2. Navigate to models > imported folder

3. Find your model's yaml/json file and adjust the settings there
4. For context length specifically, look for max_tokens or ctx_len in the file

The file shows all available parameters you can tweak, with helpful comments about valid ranges (like temperature: 0-1, etc). Just be careful editing these values - make sure they match your model's actual capabilities!

Tip: For better visibility, you can open the file with VS Code or any text editor that supports YAML syntax highlighting.

[Kep0a]

Thanks @imtuyethan !