Initial plugin design

[simonw]

It's going to let you install and run models using https://github.com/mlc-ai/mlc-llm

e.g.

from mlc_chat import ChatModule
from mlc_chat.callback import StreamToStdout
cm = ChatModule(model="Llama-2-7b-chat-hf-q4f16_1")
print(cm.generate(prompt="Three terrific names for a pet skunk"))
# Prints that out
print(cm.generate(prompt="Two more"))
# Prints out two more

[simonw]

I can't figure out how to implement conversation mode where conversations are loaded from SQLite first - I opened an issue question about that here:

• https://github.com/mlc-ai/mlc-llm/issues/737

I'll skip conversation support for the first release.

[simonw]

https://github.com/mlc-ai/notebooks/blob/main/mlc-llm/tutorial_chat_module_getting_started.ipynb is a useful tutorial.

These instructions worked for getting it installed: https://mlc.ai/mlc-llm/docs/get_started/try_out.html#get-started

[simonw]

Installing the right package for the M1/M2 requires this command:

pip install --pre --force-reinstall \
  mlc-ai-nightly \
  mlc-chat-nightly \
  -f https://mlc.ai/wheels

The llm install command doesn't yet support all of those options.

I'm going to add a llm mlc pip command as a temporary workaround for that, to ensure people install things in the correct virtual environment:

llm mlc pip install --pre --force-reinstall \
  mlc-ai-nightly \
  mlc-chat-nightly \
  -f https://mlc.ai/wheels