Open RealUnrealGameDev opened 1 year ago
I'm having similar issues when using with langchain
I got similar issues when amount of tokens (prompt + generated) were more than n_ctx (argument in Llama constructor):
model = Llama(
model_path="../../model/saiga_mistral_7b_gguf/model-q8_0.gguf",
n_gpu_layers=-1,
)
count = 0
for token in model.generate(tokens, top_k=40, top_p=0.95, temp=1.0, repeat_penalty=1.1):
count += 1
print(model.detokenize([token]))
So I found that I was getting it while generating the 513th token.
A few debug prints traced me to the place it, at first:
np.ndarray( (n_ctx, self._n_vocab), dtype=np.single )
arraySo if the error is the same for you you can limit generation size.
@abtExp
Not sure about @RealUnrealGameDev 's code
Getting a similar issue when using from langchain.embeddings import LlamaCppEmbeddings
in
File langchain/embeddings/llamacpp.py", line 113, in <listcomp>
embeddings = [self.client.embed(text) for text in texts]
^^^^^^^^^^^^^^^^^^^^^^^
File llama_cpp/llama.py", line 899, in embed
return list(map(float, self.create_embedding(input)["data"][0]["embedding"]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File llama_cpp/llama.py", line 863, in create_embedding
self.eval(tokens)
File llama_cpp/llama.py", line 543, in eval
self.input_ids[self.n_tokens : self.n_tokens + n_tokens] = batch
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: could not broadcast input array from shape (8,) into shape (0,)
Prerequisites
Expected Behavior
It should interact with the user in a chat system infinitly
Current Behavior
After a few questions and answers the program throws this error at me:
Environment and Context
I am using Llama With Persona for the chat system whilst running a GTX 1660 Ti, 16GB RAM, i7-9750H CPU, Windows 11
Failure Information (for bugs)
The error seems to be happening on Line 480, It only happens after a few messages.
Steps to Reproduce
Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.
git clone https://github.com/ngxson/llama-persona.git
pip install -r requirements.txt
python app.py
Failure Logs
This is my app.py/main.py file:
This is my config.py file: