feat: Model Pull has clear API and CLI to support Huggingface Repos

[dan-homebrew]

Goal

• cortex model pull should have clear APIs that support different model repo sources
• e.g. Huggingface, Cortex Hub

Tasklist

• [x] #1393
• [x] #1394
• [x] #1395
• [x] #1398

CLI

# Pulls immediately
cortex model pull <huggingface_url_sppecific_gguf>

# Lets user select quantization using CLI
cortex model pull <huggingface_url> 

# NOT SURE: Do we need an "info" equivalent?
# Gets repo type (e.g. GGUF, in future ONNX, TensorRT-LLM, dumps possible versions)
# Will power the "select quantization"
cortex model info <huggingface_url>
cortex model info <cortex_repo_url>    # Dumps tags

API

• Is there any change necessary to the models pull API?
• https://cortex.so/api-reference/#tag/models/post/v1/models/%7BmodelId%7D/pull

• How do we represent Huggingface strings?
• How do we handle Huggingface Repos (i.e. where user has to select quantization)?

Key Questions

• Does Cortex need an API to get Huggingface Repo metadata?
• Does this need to be bubbled to Jan, to support the Huggingface Repo page?

Linked Issues

• https://github.com/janhq/cortex.cpp/issues/1006
• https://github.com/janhq/cortex.cpp/issues/1145
• https://github.com/janhq/cortex.cpp/issues/1017
• And your PR: https://github.com/janhq/cortex.cpp/pull/1239

Jan's Requirements

1. User enters Huggingface URL in import box
2. User clicks deep link from Huggingface

Cortex should support an API, that can support the following UI:

[namchuai]

@dan-homebrew , are you sure about the command cortex model pull? Because, in cortexjs, I think we're using cortex pull. Also, ollama is using ollama pull.

One thing to note from the API: http://127.0.0.1:1337/v1/models/__MODELID__/pull. The __MODELID__ has to be put inside body of the request. This is because Drogon does not play nice if __MODELID__ contains slash.

[namchuai]

• How do we represent Huggingface strings? -> Hmm, I'm not fully understand your question. Please elaborate.

• How do we handle Huggingface Repos (i.e. where user has to select quantization)? -> There are some cases:

  • "normally", for gguf repo, all quants are placed at main branch, with same level directory.
  • cortexso will store quants in each branch.

We already handled both cases above and allowing user to select a quant that they want to use. However, for now, we only support GGUF and single GGUF file repository (multile GGUF for a model is not supported at the moment)

[namchuai]

• Does Cortex need an API to get Huggingface Repo metadata? -> Yes, mostly git API. I'm not using 3rd party lib at the moment. And I don't think there's a c++ version for the lib.

• Does this need to be bubbled to Jan, to support the Huggingface Repo page? -> I can't decide this. Let's discuss this. cc @louis-jan

[dan-homebrew]

@namchuai Can I check: can I close this issue, it seems we have implemented this already?

[namchuai]

@dan-homebrew , this one still need my last PR to support stop download. The PR is https://github.com/janhq/cortex.cpp/pull/1460 , need some testing before I mark it Ready.

[gabrielle-ong]

QAing: API for Huggingface repo

1. POST models/pull starts download 2. POST models/pull halfway shows downloaded bytes

3. DELETE models/pull stops the download

curl --location --request DELETE 'http://127.0.0.1:39281/models/pull' \
--header 'Content-Type: application/json' \
--data '{
    "taskId": <your_download_task_id>
}'

4. web sockets events emitted when model pull starts /events 5. web sockets events stopped when model pull stops /events

Pending task to add DELETE models/pull to Swagger & docs @gabrielle-ong

[gabrielle-ong]

Closing this huge epic, thanks @namchuai! Minor UX issue of stdout sent to CLI (inactive terminal), tracking in #1519