ValidationError: 1 validation error for LlamaCpp

[Jacob-Langley]

It appears to download wizardlm-13b-v1.2.Q4_K_M.gguf without issue, but then gives the below when I try to use LLM().

Same for the 7B

llm = LLM(use_larger=True, n_gpu_layers=35) Traceback (most recent call last):

Cell In[4], line 1 llm = LLM(use_larger=True, n_gpu_layers=35)

File ~\anaconda3\lib\site-packages\onprem\core.py:119 in init self.load_llm()

File ~\anaconda3\lib\site-packages\onprem\core.py:226 in load_llm self.llm = llm = LlamaCpp(

File ~\anaconda3\lib\site-packages\langchain\load\serializable.py:74 in init super().init(**kwargs)

File pydantic\main.py:341 in pydantic.main.BaseModel.init

ValidationError: 1 validation error for LlamaCpp root Could not load Llama model from path: C:\Users\jlangley\onprem_data\wizardlm-13b-v1.2.Q4_K_M.gguf. Received error fileno (type=value_error)

[amaiya]

What version of llama-cpp-python are you using?

Does the error still occur if you supply verbose=True to LLM?

llm = LLM(use_larger=True, n_gpu_layers=35, verbose=True)

[Jacob-Langley]

verbose=True appears to have done the trick! -- (why?)

[amaiya]

Due to what seems to be a bug in llama-cpp-python.