Feedback on `Run Llama 2 Locally with Python`

[stbennet]

Regarding https://swharden.com/blog/2023-07-29-ai-chat-locally-with-python/

I had some issues getting your method to work on my computer. I ended up getting it to work by swapping the ggmlv3.q8_0.bin file for a .gguf file which resolved the issue.

The full error message I receive when running your code is:

gguf_init_from_file: invalid magic characters 'tjgg'
error loading model: llama_model_loader: failed to load model from ./llama-2-7b-chat.ggmlv3.q8_0.bin

llama_load_model_from_file: failed to load model
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |
Traceback (most recent call last):
  File "D:\Users\benne\Documents\Python\llama\pilot-playing-swharden.py", line 5, in <module>
    LLM = Llama(model_path="./llama-2-7b-chat.ggmlv3.q8_0.bin")
  File "D:\Users\benne\Documents\Python\llama\.venv\lib\site-packages\llama_cpp\llama.py", line 957, in __init__
    self._n_vocab = self.n_vocab()
  File "D:\Users\benne\Documents\Python\llama\.venv\lib\site-packages\llama_cpp\llama.py", line 2264, in n_vocab
    return self._model.n_vocab()
  File "D:\Users\benne\Documents\Python\llama\.venv\lib\site-packages\llama_cpp\llama.py", line 252, in n_vocab
    assert self.model is not None
AssertionError

Thanks for the guide, I've been enjoying playing around with the llama model.