Open QuangBK opened 9 months ago
same problem, but it happens in Chinese
Inference in Chinese doesn't work for me, +1
Inference in Chinese doesn't work for me, +1
+1. Is there any solution?
I've been investigating the issue and it seems that the problem might be with the bytes tokenizer using trie. From what I understand, the method _tokenize_prefix is designed to return the longest valid bytes token. However, the assumption underlying the bytes tokenizer, as implemented in this repository, does not seem to hold for all tokenizers.
To address this, my suggestion is to switch to the default tokenizer. The following code snippet outlines the proposed change:
def _tokenize_prefix(self, prompt):
if isinstance(prompt, bytes):
prompt = prompt.decode("utf-8")
return self._orig_tokenizer(prompt).input_ids, []
This modification ensures that if the prompt is in bytes, it gets decoded to a UTF-8 string before tokenization. This approach might be more robust and universally applicable.
Same thing happening to me in Korean.
@anhvth I can't seem to find _orig_tokenizer from the source code. Where Can I find it? Additionally, guaranteeing that prompt is decoded into bytes using utf-8 seems to have already been implemented:
def __call__(self, grammar, max_tokens=1000000, n=1, top_p=1, temperature=0.0, ensure_bos_token=True):
assert n == 1, "Still need to add support for n > 1!"
# get our current context in bytes
prompt = self._current_prompt()
prompt = bytes(prompt, encoding="utf-8")
Same problem, I've left a comment on a related ticket: https://github.com/guidance-ai/guidance/issues/454#issuecomment-1878149397
update: using llamacpp instead of transformers solved the problem for me.
@MINGYUK good to know thanks, unfortunately llama.cpp doesn't work with gptq
I had the same problem when using Japanese in the prompts. However, after reflecting the commit 8f5b3bdfe28455ef267da3e0e590a0d9a4d08104, the error disappeared. I don't yet understand enough to explain the details, but I shared the information for your reference. I hope it helps someone.
The bug Thank you for the new version! I'm working with Llama-2-13B-chat. While it works fine in English. When I added some non-English sentences to the prompt, it gave this error as below.
To Reproduce
My model and prompt with non-English sentences.
When I use Guidance
It gives this error.
However, without Guidance, it works well as below:
System info (please complete the following information):
guidance.__version__
): 0.1.3