Initial plugin design

[simonw]

I want to use https://github.com/abetlen/llama-cpp-python to run GGML models from https://huggingface.co/TheBloke

I'm going to start with these two:

• https://huggingface.co/TheBloke/Llama-2-7B-GGML/blob/main/llama-2-7b.ggmlv3.q8_0.bin
• https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/main/llama-2-7b-chat.ggmlv3.q8_0.bin

[simonw]

I got llama-cpp-python to install correctly on my M2 laptop which has XCode:

pip install llama-cpp-python

Then I found the wheel file for it:

cd ~/Library/Caches/pip/wheels
find . | grep llama

Here it is:

ls -lah ./e2/67/cb/481cfaabbb5fd5edab627c5b475de63e1b6f7d4d7b678d4d25/llama_cpp_python-0.1.77-cp311-cp311-macosx_13_0_arm64.whl

231K

Then I uploaded it here:

https://static.simonwillison.net/static/2023/llama_cpp_python-0.1.77-cp311-cp311-macosx_13_0_arm64.whl

And now this seems to work in a fresh virtual environment, without needing a C compiler (hopefully):

pip install https://static.simonwillison.net/static/2023/llama_cpp_python-0.1.77-cp311-cp311-macosx_13_0_arm64.whl

[simonw]

I'm going to try having three installation options.

This one leaves you to install llama-cpp-python independently:

llm install llm-llama-cpp

This one will depend directly on llama-cpp-python and try to compile it:

llm install 'llm-llama-cpp[compile]'

This one will download that wheel file I made:

llm install 'llm-llama-cpp[mac_arm64]'

[simonw]

This seemed to work in dev!

pip install -e '.[mac_arm64,test]'

Obtaining file:///Users/simon/Dropbox/Development/llm-llama-cpp
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting llama-cpp-python@ https://static.simonwillison.net/static/2023/llama_cpp_python-0.1.77-cp311-cp311-macosx_13_0_arm64.whl (from llm-llama-cpp==0.1)
...

From this pyproject.toml:

[project]
name = "llm-llama-cpp"
version = "0.1"
description = "LLM plugin for running models using llama.cpp"
readme = "README.md"
authors = [{name = "Simon Willison"}]
license = {text = "Apache-2.0"}
classifiers = [
    "License :: OSI Approved :: Apache Software License"
]
dependencies = [
    "llm"
]

[project.urls]
Homepage = "https://github.com/simonw/llm-llama-cpp"
Changelog = "https://github.com/simonw/llm-llama-cpp/releases"
Issues = "https://github.com/simonw/llm-llama-cpp/issues"
CI = "https://github.com/simonw/llm-llama-cpp/actions"

[project.entry-points.llm]
llama_cpp = "llm_llama_cpp"

[project.optional-dependencies]
test = ["pytest"]
mac_arm64 = [
  "llama-cpp-python @ https://static.simonwillison.net/static/2023/llama_cpp_python-0.1.77-cp311-cp311-macosx_13_0_arm64.whl"  
]