The example for tokenization assumes that mlx.data.tokenizer_helpers.read_trie_from_spm just returns a CharTrie, however it also returns weights. I updated the docs to reflect that it returns a tuple. Feel free to remove the piece where I pass the weights along to the Tokenizer, I'm not sure if that's correct.
The example for tokenization assumes that
mlx.data.tokenizer_helpers.read_trie_from_spmjust returns a CharTrie, however it also returns weights. I updated the docs to reflect that it returns a tuple. Feel free to remove the piece where I pass the weights along to the Tokenizer, I'm not sure if that's correct.