This PR reimplements RegexTokenizer._encode_chunk() using dynamic programming to return a guaranteed optimal (minimum number of tokens) tokenization of a chunk.
After training, the vocabulary is fixed. During encoding, _encode_chunk() is called for each chunk to get a tokenization of it. Currently, _encode_chunk() uses something similar to the BPE algorithm used in the training (but with the fixed vocabulary) to determine the tokens. However, this approach is slow and does not guarantee an optimal tokenization of the chunk.
Tested on a 1MB text from wikitext_103, the encoding is more than 20% faster, and the compression is improved by about 0.5%.
This PR reimplements RegexTokenizer._encode_chunk() using dynamic programming to return a guaranteed optimal (minimum number of tokens) tokenization of a chunk.
After training, the vocabulary is fixed. During encoding, _encode_chunk() is called for each chunk to get a tokenization of it. Currently, _encode_chunk() uses something similar to the BPE algorithm used in the training (but with the fixed vocabulary) to determine the tokens. However, this approach is slow and does not guarantee an optimal tokenization of the chunk.
Tested on a 1MB text from wikitext_103, the encoding is more than 20% faster, and the compression is improved by about 0.5%.