Discussion: Cortex.cpp Model and model.yaml

[dan-homebrew]

Overview

• Linked: #1040
• Where and how do we store models?
• How do we structure model folders?
• How do we detect models?
• What is the format of our model.yaml?

Docs

• https://cortex.so/docs/model-yaml
• https://github.com/janhq/jan/issues/2843
• https://github.com/janhq/jan/issues/3251 (poorly scoped)

[0xSage]

Data Folder Questions

1. What is the data structure of ~/.cortexcpp/models? Where do the following go

   • model yaml
   • model binaries, especially of multiple binaries (model_1_of_5.bin)
   • versions of the same model, e.g. llama3.1, llama3.2
   • presets (if any, can also defer for later discussion)

2. Is our preference on a more flat folder structure? rather than super nested?

3. Previously, we had many bugs resulting from expecting folder & file names to be a certain way. e.g. we expected unique_model_id (used by backend) to be the same as the model folder name Something to be aware of in this iteration 🙏

Model Downloading

1. What happens when model download fails halfway (e.g. internet disconnected)?
2. How do we detect models? e.g. if users "import models locally" would it still work
3. How do we version models?
   • If we update our model.yaml, or remote model binary in the HF branch, will download still work?
   • Or will "redownloading/updating" fail due to "model exists"
4. Letting users do cortex models update is currently out of scope right?

Model importing

1. Can users import existing models?
2. Is it hard copy or symlink

Model YAML

1. We are auto populating the model.yaml if user downloads a new GGUF file (not from our HF repo)?
2. What happens when user deletes the YAML accidentally?
3. What happens when YAML is there but Binary is deleted? (should we spec little unit tests like this)
4. Is this up to date? https://cortex.so/docs/model-yaml/
5. If users update YAMLs, when do changes take effect?

[vansangpfiev]

Data Folder Questions

Q 1. What is the data structure of ~/.cortexcpp/models?

~/.cortexcpp/
|___ models
       |__ tinyllama.yaml
       |__ tinyllama
       |     |__ model_01.gguf
       |     |__ model_02.gguf
       |     |__ model.yml
       |__ llama3.1
       |__ llama3.2

After downloading model (from cortexso or other HF repository), cortex generates tinyllama.yaml file which is used for model management.

Q 2. Is our preference on a more flat folder structure? rather than super nested? Can you give an example of the flat folder structure? Q 3. Can you elaborate more about the issue? Do we have any ticket to track that issue yet?

Model importing Q 1. Can users import existing models?

• We don't support it yet. TBD when we will support it. Q 2. Is it hard copy or symlink
• It will be a symlink

cc: @0xSage

[nguyenhoangthuan99]

Model Yaml

1. Folder structure:

   ~/.cortexcpp/
   |___ models
      |__ tinyllama.yaml
      |__ tinyllama
      |     |__ model_01.gguf
      |     |__ model_02.gguf
      |     |__ model.yml
      |__ llama3.1
      |__ llama3.2

2. With model not from cortexso:

   • When model is downloaded, it will parse and save information to <model_id>/model.yml and <model_id>.yaml, this 2 files is the same.
   • When user delete 1 of 2 .yaml file. We can provide a command to recover it like cortex-cpp models recover, to check and resolve yml error.

3. when YAML is there but binary is deleted, will raise No such file or directory error when load models.

4. The doc from https://cortex.so/docs/model-yaml/ is up to date.

5. To apply update user need to stop running models and re run chat to load new configuration

[namchuai]

Model Downloading

1. What happens when model download fails halfway (e.g. internet disconnected)?

   • We don't support resume failed/pause download.
   • If model download is failed, its <model_id>.yaml (inside /models) won't be created and won't display in our model list.

2. How do we detect models? e.g. if users "import models locally" would it still work

   • Scan for <model_id>.yaml file inside /models.

3. How do we version models?

   • Currently, we using a field version inside yaml file to store the model's version. Please not that we don't have logic to support model update at the moment. For now, Version is just for display purpose.

3.1. If we update our model.yaml, or remote model binary in the HF branch, will download still work? Or will "redownloading/updating" fail due to "model exists"

• It will display model exists.

4. Letting users do cortex models update is currently out of scope right?
   • Yes, it is. We haven't work on this.