bug: What's wrong with `tinyllama-1.1b`?

[imtuyethan]

1. Start the first conversation in Jan
2. Choose tinyllama-1.1b
3. It started generating nonsense. What's wrong with tinyllama-1.1b? Is it because of the model or because of our configuration?

https://github.com/user-attachments/assets/f8a6eed0-58fd-4a1a-a779-3e7ad5e75675

[Van-QA]

Need to review the stop token and promt template.

[nguyenhoangthuan99]

• [x] Chat template:

1. Currently we are using this chat template: <|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>, but when I use jinja2 tool to render chat template from source it should look like this \n\n<|system|>\n{system_message}</s>\n\n\n\n\n<|user|>\n{prompt}</s>\n\n\n<|assistant|>\n\n.
2. I updated it in our repo. And test again but it still generate answer not related to question.

• [x] stop token is correct </s>. It is infer from here
• [x] Confirm with original source https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.6/blob/main/ggml-model-q4_0.gguf

I used model from above source with the correct prompt template + stop token, it still have buggy answer. I think this bug can be caused by model's quality.

• [x] Test with original llamacpp server, tinyllama still showing buggy answer

[0xSage]

thanks for the thorough investigation @nguyenhoangthuan99 🙏 closing as its likely a tinyllama quality issue at this point.