simonw / llm-mlc

LLM plugin for running models using MLC
Apache License 2.0
174 stars 8 forks source link

Mechanism for setting aliases for models #4

Closed simonw closed 11 months ago

simonw commented 11 months ago

Maybe a -a/--alias option that can be passed zero or more times to the llm mlc download-model command.

It would be useful to apply aliases later as well, but I think that's a feature that should go in LLM itself.

simonw commented 11 months ago

I'm going to depend on LLM 0.7 or higher and have the download-model ... --alias llama2 option work by writing to the aliases.json file.

simonw commented 11 months ago

I just noticed that this:

llm mlc download-model Llama-2-7b-chat

Results in a model activated like this:

llm -m mlc-chat-Llama-2-7b-chat-hf-q4f16_1 \
  'five names for a cute pet ferret'

It should set up Llama-2-7b-chat as an alias automatically too.

simonw commented 11 months ago

Updated documentation: https://github.com/simonw/llm-mlc/blob/7b743ad54430ef44d84be8a398e869481c84c328/README.md#installing-models