search

tairov / llama2.mojo

Inference Llama 2 in one file of pure 🔥
https://www.modular.com/blog/community-spotlight-how-i-built-llama2-by-aydyn-tairov
MIT License
2.09k stars 140 forks source link

How to create my own tokenizer.bin? #7

Closed hungtooc closed 1 year ago

hungtooc commented 1 year ago

as title

tairov commented 1 year ago

check this out : https://github.com/karpathy/llama2.c/blob/master/tokenizer.py

hungtooc commented 1 year ago

check this out : https://github.com/karpathy/llama2.c/blob/master/tokenizer.py

Thank you, I still don't understand how to use my prompt, which is no longer "once upon a time, there was a little girl named Lily."

tairov commented 1 year ago

@hungtooc I haven't implemented prompt method yet. Will do it soon this week.