Open Liufeiran123 opened 3 months ago
It seems that the problem with the Qwen1.5-7B model was actually a problem in llama.cpp that has already been fixed (https://github.com/ggerganov/llama.cpp/issues/7384), so i guess that this problem may be closed.
model.tokenize result is different from tranformers tokenizer result
using the same mode Qwen1.5-7B input_ids1 = model.tokenize(prompt.encode("utf-8"))
input_ids2 = tokenizer([prompt], padding=False)["input_ids"]
input_ids1 != input_ids2