Feedback on `Run Llama 2 Locally with Python`

[lingvisa]

Regarding https://swharden.com/blog/2023-07-29-ai-chat-locally-with-python/

Hi, thank you for the tutorial on llma2. I copied your code but received this error message. I am using MacPro M1 Max laptop. It looks like the loading of the model failed, but no meaningful error message.

Connected to pydev debugger (build 231.9011.38) gguf_init_from_file: invalid magic number 67676a74 error loading model: llama_model_loader: failed to load model from ./llama-2-7b-chat.ggmlv3.q8_0.bin

llama_load_model_from_file: failed to load model Traceback (most recent call last): File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1496, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/Users/martin/nlp/gpt/dlai_course/llma2/llama2_test.py", line 3, in LLM = Llama(model_path='./llama-2-7b-chat.ggmlv3.q8_0.bin') File "/Users/martin/miniconda3/envs/llama/lib/python3.9/site-packages/llama_cpp/llama.py", line 365, in init assert self.model is not None AssertionError python-BaseException

Process finished with exit code 1

[swharden]

Hi @lingvisa, thanks for reporting this!

On the llama-cpp-python page https://pypi.org/project/llama-cpp-python/ it has a note for M1 users

Note: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh

If this resolves your issue let me know and I can add a note to the webpage! Otherwise I'm not sure what to recommend other than googling the error message and finding how other mac users use that python page.

Hope it helps! Scott