abetlen / llama-cpp-python

Python bindings for llama.cpp
https://llama-cpp-python.readthedocs.io
MIT License
8.16k stars 970 forks source link

Fix: Refactor Batching notebook to use new sampler chain API #1793

Open lukestanley opened 1 month ago

lukestanley commented 1 month ago
lukestanley commented 1 month ago

I was trying to figure out what regressed with the low level API, and I saw that there was a bit more recent batching notebook. But the Batching notebook itself was broken so I fixed it.

I had to diff changes in llama_cpp.py to find the API changes.

To use GitHub Codespaces after making a Codespace from the branch, I had to:

git submodule update --init --recursive
pip install --upgrade pip
pip install -e .
wget "https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q2_K.gguf?download=true"
mv "mistral-7b-v0.1.Q2_K.gguf?download=true" "mistral-7b-v0.1.Q2_K.gguf"

When Codespace offers to install the Jupyter extension, I accepted it. I am then able to run the whole notebook.