dalgibbard
commented
2 months ago I'm seeing a possibly related error; when loading the cached embeddings, it states:
Creating index from embeddings...
Loading local LLM model...
LLM context size: 9200Then when asking a question, it started writing out the answer, and then after a while returns:
(generating contextual guidance...)Traceback (most recent call last):
File "/home/user/.local/bin/dir-assistant", line 8, in <module>
sys.exit(main())
File "/home/user/.local/lib/python3.10/site-packages/dir_assistant/main.py", line 153, in main
start(args, config_dict["DIR_ASSISTANT"])
File "/home/user/.local/lib/python3.10/site-packages/dir_assistant/cli/start.py", line 166, in start
llm.stream_chat(user_input)
File "/home/user/.local/lib/python3.10/site-packages/dir_assistant/assistant/git_assistant.py", line 101, in stream_chat
super().stream_chat(user_input)
File "/home/user/.local/lib/python3.10/site-packages/dir_assistant/assistant/base_assistant.py", line 132, in stream_chat
stream_output = self.run_stream_processes(user_input, True)
File "/home/user/.local/lib/python3.10/site-packages/dir_assistant/assistant/cgrag_assistant.py", line 98, in run_stream_processes
output_history = self.run_completion_generator(cgrag_generator, output_history, False)
File "/home/user/.local/lib/python3.10/site-packages/dir_assistant/assistant/llama_cpp_assistant.py", line 49, in run_completion_generator
for chunk in completion_output:
File "/home/user/.local/lib/python3.10/site-packages/llama_cpp/llama_chat_format.py", line 289, in _convert_text_completion_chunks_to_chat
for i, chunk in enumerate(chunks):
File "/home/user/.local/lib/python3.10/site-packages/llama_cpp/llama.py", line 1216, in _create_completion
for token in self.generate(
File "/home/user/.local/lib/python3.10/site-packages/llama_cpp/llama.py", line 808, in generate
self.eval(tokens)
File "/home/user/.local/lib/python3.10/site-packages/llama_cpp/llama.py", line 658, in eval
self.scores[n_past + n_tokens - 1, :].reshape(-1)[::] = logits
IndexError: index 9200 is out of bounds for axis 0 with size 9200Is this the same issue?
curvedinf
When chatting with an LLM, sometimes dir-assistant sends too much context to the llm.
Counts appear correct on dir-assistant's end. Perhaps this is because in some cases, the embedding models's token set differs from the chat llm's, so it can't produce correct token counts.
We should probably use the chat model's token counter if available, and only fall back to the embedding model if they aren't available for some reason (for instance an API doesn't offer a token counter).
For API models, you can simply set the context limit lower than the API's limit as a user. However, for local models this isn't possible because the context limit also sets the memory allocated.