search

RWKV / rwkv.cpp

INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
MIT License
1.37k stars 90 forks source link

Add support for the world tokenizer #86

Closed Mathmagician8191 closed 1 year ago

Mathmagician8191 commented 1 year ago

Tokenizer implementation taken from https://github.com/BlinkDL/ChatRWKV/tree/main/tokenizer with test code removed.

This pull request adds a tokenizer command line argument to chat_with_bot.py, generate_completions.py and measure_pexplexity.py. Current options are the original 20B tokenizer (default) and the new world tokenizer.

Mathmagician8191 commented 1 year ago

Everything should be fixed now except for not having any tests

Mathmagician8191 commented 1 year ago

There is now a test verifying both encoding and decoding of the main test string from the original implementation

saharNooby commented 1 year ago

Let me know what do you think about this, and I'll merge.

Mathmagician8191 commented 1 year ago

The long test string should be fine, the test isn't likely to be run that often and it makes sure all edge cases are handled