danielgross / localpilot

MIT License
3.34k stars 143 forks source link

Add 13B and 4_0 model for metal #20

Open siilats opened 11 months ago

siilats commented 11 months ago

use

pip uninstall llama-cpp-python -y
CMAKE_ARGS="-DLLAMA_METAL=on" pip install -U llama-cpp-python --no-cache-dir
pip install 'llama-cpp-python[server]'

to install metal version. only works on 4_0 quant models