OSError: [WinError -529697949] Windows Error 0xe06d7363

[zhiyixu]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [x] I carefully followed the README.md.
• [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I find some issues with the same problem, but the solutions in the issue do not work with my computer.
• [x] I reviewed the Discussions, and have a new bug or useful enhancement to share.
  • i searched 'Llama' object has no attribute 'ctx' in the the serach box, nothing returns.

Expected Behavior

Please provide a detailed written description of what you were trying to do, and what you expected llama-cpp-python to do. i want to run the llama model with code below

from llama_cpp import Llama
llm = Llama(model_path="D:/LLMs/llama.cpp/zh-models/7B/ggml-model-q4_0.bin",n_threads=4)
output = llm("give me some tips about english learning.", max_tokens=256, stop=["Q:", "\n"], echo=True)
print(output)

Current Behavior

Please provide a detailed written description of what llama-cpp-python did, instead. after i run the commad above, it give me the errors ilke:

(env_llm) PS C:\Users\xxx> python .\llama-cpp.py
llama.cpp: loading model from D:/LLMs/llama.cpp/zh-models/7B/ggml-model-q4_0.bin
Traceback (most recent call last):
  File "C:\Users\xxxx\llama-cpp.py", line 2, in <module>
    llm = Llama(model_path="D:/LLMs/llama.cpp/zh-models/7B/ggml-model-q4_0.bin",n_threads=4)
  File "D:\software\anaconda\envs\env_llm\lib\site-packages\llama_cpp\llama.py", line 193, in __init__
    self.ctx = llama_cpp.llama_init_from_file(
  File "D:\software\anaconda\envs\env_llm\lib\site-packages\llama_cpp\llama_cpp.py", line 262, in llama_init_from_file
    return _lib.llama_init_from_file(path_model, params)
OSError: [WinError -529697949] Windows Error 0xe06d7363
Exception ignored in: <function Llama.__del__ at 0x00000254E26F5A60>
Traceback (most recent call last):
  File "D:\software\anaconda\envs\env_llm\lib\site-packages\llama_cpp\llama.py", line 1333, in __del__
    if self.ctx is not None:
AttributeError: 'Llama' object has no attribute 'ctx'
(env_llm) PS C:\Users\xxxx>

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu	Name	NumberOfCores	NumberOfLogicalProcessors	ProcessorId
13th Gen Intel(R) Core(TM) i5-13400	10	16	BFEBFBFF000B06F2

• Operating System, e.g. for Linux:

$ uname -a Microsoft Windows 10 Enterprise

• SDK version, e.g. for Linux:

$ python3 --version -> `3.9.10`
$ make --version -> `cmake version 3.26.4`
$ g++ --version -> i don't have this, is that the reason why the error occured? 

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template. error goes like this, i am not sure it' a bug or not

(env_llm) PS C:\Users\xxx> python .\llama-cpp.py
llama.cpp: loading model from D:/LLMs/llama.cpp/zh-models/7B/ggml-model-q4_0.bin
Traceback (most recent call last):
  File "C:\Users\xxxx\llama-cpp.py", line 2, in <module>
    llm = Llama(model_path="D:/LLMs/llama.cpp/zh-models/7B/ggml-model-q4_0.bin",n_threads=4)
  File "D:\software\anaconda\envs\env_llm\lib\site-packages\llama_cpp\llama.py", line 193, in __init__
    self.ctx = llama_cpp.llama_init_from_file(
  File "D:\software\anaconda\envs\env_llm\lib\site-packages\llama_cpp\llama_cpp.py", line 262, in llama_init_from_file
    return _lib.llama_init_from_file(path_model, params)
OSError: [WinError -529697949] Windows Error 0xe06d7363
Exception ignored in: <function Llama.__del__ at 0x00000254E26F5A60>
Traceback (most recent call last):
  File "D:\software\anaconda\envs\env_llm\lib\site-packages\llama_cpp\llama.py", line 1333, in __del__
    if self.ctx is not None:
AttributeError: 'Llama' object has no attribute 'ctx'
(env_llm) PS C:\Users\xxxx>

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. step 1: install this package with pip install --upgrade --force-reinstall --no-cache-dir llama-cpp-python i found this cmd in other related issue.
2. step 2 cpoy the model path with right click "D:\LLMs\llama.cpp\zh-models\7B\ggml-model-q4_0.bin"
3. step 3 run the script above like

   from llama_cpp import Llama
   llm = Llama(model_path="D:/LLMs/llama.cpp/zh-models/7B/ggml-model-q4_0.bin",n_threads=4)
   output = llm("give me some tips about english learning.", max_tokens=256, stop=["Q:", "\n"], echo=True)
   print(output)

4. find the error.

Note: Many issues seem to be regarding functional or performance issues / differences with llama.cpp. In these cases we need to confirm that you're comparing against the version of llama.cpp that was built with your python package, and which parameters you're passing to the context.

Try the following:

1. git clone https://github.com/abetlen/llama-cpp-python
2. cd llama-cpp-python
3. rm -rf _skbuild/ # delete any old builds
4. python setup.py develop
5. cd ./vendor/llama.cpp
6. Follow llama.cpp's instructions to cmake llama.cpp
7. Run llama.cpp's ./main with the same arguments you previously passed to llama-cpp-python and see if you can reproduce the issue. If you can, log an issue with llama.cpp

Failure Logs

in step4 the errors occured like this, but i don't know where to find the CMake's output so i paste some output in cmd

An error occurred while configuring with CMake.
  Command:
    'C:\Program Files\CMake\bin\cmake.EXE' 'D:\LLMs\llama-cpp-python' -G Ninja -D_SKBUILD_FORCE_MSVC=1930 --no-warn-unused-cli '-DCMAKE_INSTALL_PREFIX:PATH=D:\LLMs\llama-cpp-python\_skbuild\win-amd64-3.9\cmake-install' -DPYTHON_VERSION_STRING:STRING=3.9.0 -DSKBUILD:INTERNAL=TRUE '-DCMAKE_MODULE_PATH:PATH=D:\software\anaconda\envs\env_llm\lib\site-packages\skbuild\resources\cmake' '-DPYTHON_EXECUTABLE:PATH=D:\software\anaconda\envs\env_llm\python.exe' '-DPYTHON_INCLUDE_DIR:PATH=D:\software\anaconda\envs\env_llm\Include' '-DPYTHON_LIBRARY:PATH=D:\software\anaconda\envs\env_llm\libs\python39.lib' '-DPython_EXECUTABLE:PATH=D:\software\anaconda\envs\env_llm\python.exe' '-DPython_ROOT_DIR:PATH=D:\software\anaconda\envs\env_llm' -DPython_FIND_REGISTRY:STRING=NEVER '-DPython_INCLUDE_DIR:PATH=D:\software\anaconda\envs\env_llm\Include' '-DPython_LIBRARY:PATH=D:\software\anaconda\envs\env_llm\libs\python39.lib' '-DPython_NumPy_INCLUDE_DIRS:PATH=D:\software\anaconda\envs\env_llm\lib\site-packages\numpy\core\include' '-DPython3_EXECUTABLE:PATH=D:\software\anaconda\envs\env_llm\python.exe' '-DPython3_ROOT_DIR:PATH=D:\software\anaconda\envs\env_llm' -DPython3_FIND_REGISTRY:STRING=NEVER '-DPython3_INCLUDE_DIR:PATH=D:\software\anaconda\envs\env_llm\Include' '-DPython3_LIBRARY:PATH=D:\software\anaconda\envs\env_llm\libs\python39.lib' '-DPython3_NumPy_INCLUDE_DIRS:PATH=D:\software\anaconda\envs\env_llm\lib\site-packages\numpy\core\include' -DCMAKE_BUILD_TYPE:STRING=Release
  Source directory:
    D:\LLMs\llama-cpp-python
  Working directory:
    D:\LLMs\llama-cpp-python\_skbuild\win-amd64-3.9\cmake-build
Please see CMake's output for more information.

i have a llama.cpp installed before, i run with cmd below

.\bulid\bin\Release\main -m .\zh-models\7B\ggml-model-q4_0.bin --color -f prompts/alpaca.txt -ins -c 2048 --temp 0.2 -n 256 --repeat_penalty 1.3

it worked well with no error.

Please include any relevant log snippets or files. If it works under one configuration but not under another, please provide logs for both configurations and their corresponding outputs so it is easy to see where behavior changes.

Also, please try to avoid using screenshots if at all possible. Instead, copy/paste the console output and use Github's markdown to cleanly format your logs for easy readability.

Example environment info: llama-cpp-python$ git log | head -1

commit 3977eea8e984c394a3c6d982a7c00804beb67a29 (HEAD -> main, origin/main, origin/HEAD)
Merge: 71f4582 30d32e9
Author: Andrei <abetlen@gmail.com>
Date:   Fri Jun 2 13:02:48 2023 -0400

    Merge pull request #310 from gjmulder/auto-docker

    Auto docker v2 - dockerised Open Llama 3B image w/OpenBLAS enabled server

llama-cpp-python$ python3 --version Python 3.9.0

llama-cpp-python$ pip list | egrep "uvicorn|fastapi|sse-starlette|numpy" fastapi 0.96.0 numpy 1.24.3 sse-starlette 1.6.1 uvicorn 0.22.0

llama-cpp-python/vendor/llama.cpp$ git log | head -3

commit 3977eea8e984c394a3c6d982a7c00804beb67a29 (HEAD -> main, origin/main, origin/HEAD)
Merge: 71f4582 30d32e9
Author: Andrei <abetlen@gmail.com>
Date:   Fri Jun 2 13:02:48 2023 -0400

i don't know it's match all your requests, but i have try to follow all the tip in this template.

[gjmulder]

Firstly, thanks for all the info. It makes trying to understand this bug easier. A number of people have reported it on Windows, and it could either be:

1. A Windows compilation issue
2. A model versioning issue

i have a llama.cpp installed before, i run with cmd below

This is valuable as we can verify the version of the model! Please provide the output from main.exe. It should report the ggml version of the model and also print the commit tag of llama.cpp. From the commit tag I might be able to determine which version of llama.cpp you compiled.

My suspicion is that you're using a ggml v2 model with an old main.exe, but when you use a new version of llama.cpp via llama-cpp-python it can't read the ggml v2 model data.

[zhiyixu]

@gjmulder you are sooooooo right about this problem. i pull the latest llama.cpp repo and quantize the model again, then it worked!
thx!

[ahlwjnj]

Could you please share some tips about run python API on Windows: How can we generate the "llama.dll" file when there is no "make" tool in Windows? Cause hard to inst