Closed francis2tm closed 4 months ago
Currently, there is a coupling of endpoints and model type (model "kind" in our code):
The second issue is trivial to solve: just pass the endpoint explicitly to preload and the problem is solved (we do a similar thing in integration tests).
The first issue causes more headaches. Possible solutions:
Here's my solution: provide a file of the form
[llama.cpp]
neural-chat
phi
TinyLlama
zephyr
nomic-embed
mistral
...
[whisper.cpp]
distill
whisper
...
The file lives in the models directory (i.e. data_dir/models
).
Expert users can and are encouraged to add their own models to the list.
We can later extend the format to add additional information. But that's for the future.
Whenever we pass a request to a model endpoint, we first find out, which endpoint we need:
It looks promising. However, I have a few concerns with this:
There are already a lot of models that llama.cpp supports (4k+). Could we do like:
[llama.cpp]
*.gguf
Or any model from huggingface that has the tag GGUF
. (https://huggingface.co/models?library=gguf)
Upgrading - what happens if someone has an old version and there's an update which should cause the file to be updated?
Implementation:
data_dir/models
Overview
In the future, we want to support multiple ML backends for each endpoint
Example:
chat/completions
can use:audio/transcriptions
can use:images/generations
can use: