janhq / cortex.cpp

Run and customize Local LLMs.
https://cortex.so
Apache License 2.0
1.97k stars 111 forks source link

idea: standardize <engine_id> across `get` and `list` #1283

Closed 0xSage closed 3 weeks ago

0xSage commented 1 month ago

Problem Statement

  1. engines list vs
  2. engine get
  3. have inconsistent engine IDs
❯ sudo cortex-nightly engines list
+---+--------------+-------------------+---------+--------------+
| # | Name         | Supported Formats | Version | Status       |
+---+--------------+-------------------+---------+--------------+
| 1 | ONNXRuntime  | ONNX              | 0.0.1   | Incompatible |
+---+--------------+-------------------+---------+--------------+
| 2 | llama.cpp    | GGUF              | 0.0.1   | Ready        |
+---+--------------+-------------------+---------+--------------+
| 3 | TensorRT-LLM | TensorRT Engines  | 0.0.1   | Incompatible |
+---+--------------+-------------------+---------+--------------+

❯ sudo cortex-nightly engines get ONNXRuntime
A subcommand is required
Run with --help for more information.

❯ sudo cortex-nightly engines get cortex.ONNXRuntime
A subcommand is required
Run with --help for more information.

❯ sudo cortex-nightly engines get cortex.onnx
+-------------+-------------------+---------+--------------+
| Name        | Supported Formats | Version | Status       |
+-------------+-------------------+---------+--------------+
| ONNXRuntime | ONNX              | 0.0.1   | Incompatible |
+-------------+-------------------+---------+--------------+

Related

1168

Questions

1.. For engine IDs, what are we going with?

  1. Can we ensure that the IDs displayed by engines list are consistent with the IDs used in engines get <ID>?
  2. Possibly terrible idea: Should we let users get by index? e.g. cortex engines get 0

Current

0xSage commented 1 month ago

cc @dan-homebrew can you take a look. Do we need to fix the current engine naming?

dan-homebrew commented 3 weeks ago

This is linked to #1168.

gabrielle-ong commented 3 weeks ago

Closing as duplicate of #1168