search

janhq / cortex.cpp

Local AI API Platform
https://cortex.so
Apache License 2.0
2.06k stars 116 forks source link

feat: `cortex engines list` #1074

Closed vansangpfiev closed 1 month ago

vansangpfiev commented 2 months ago

command: cortex-cpp engines list

nguyenhoangthuan99 commented 2 months ago

When building the binary, in window I met a problem that doesn't appear on Linux and Mac:

Image

To resolve this please check this commit if you meet this error in the future

nguyenhoangthuan99 commented 2 months ago

to use commands: ./cortex-cpp engines list

It will show result like this

Image

The onnx engine'status will be not_supported in Linux and Mac. and if the engine was found the status will be ready otherwise not_initialized

dan-homebrew commented 2 months ago

@nguyenhoangthuan99 @vansangpfiev I recommend for us to simplify the cortex engines list to a simpler table

# Name Supported Formats Version Status
1 ONNXRuntime ONNX 0.0.1 Incompatible
2 llama.cpp GGUF 0.0.1 ready
3 TensorRT-LLM TensorRT Engines 0.0.1 Not installed

I personally think the description and product_name is a bit extraneous, should keep this simple

For the TensorRT-LLM supported format, I would need to ask you guys what the "Supported Format" is - .engine files? (not sure)

dan-homebrew commented 1 month ago

@namchuai Can I reassign this to you, since you are the main owner of engines API endpoint and CLI

namchuai commented 1 month ago

@dan-homebrew yes, I will work on this. However, I have question regarding the semantic naming for engine which we have a discussion here: https://github.com/janhq/cortex.cpp/issues/1168

gabrielle-ong commented 1 month ago

QAed with #1073 (see screenshots there)

Linked to open followup issue on #1168