search

daquexian / faster-rwkv

123 stars 11 forks source link

How can i change default vocabs for world model? #4

Open xiaol opened 1 year ago

xiaol commented 1 year ago

need some change for default vocabs file

like this one, add some new special tokens in the end of file:

https://huggingface.co/xiaol/RWKV-paper-reviewer-and-reading-comprehension-wenda-Worldv4-7B-16k/blob/main/rwkv_vocab_v20230424.txt

daquexian commented 1 year ago

try https://github.com/daquexian/faster-rwkv/blob/master/tools/convert_tokenizer.py to generate the faster-rwkv tokenizer model from vocab file.