Closed sudoskys closed 6 months ago
Refactored LLMTokenizer class methods to improve readability and maintainability.
Disabled initialization of NERDSTASH TOKENIZER as it causes errors on non-English hostnames or paths with spaces: https://github.com/google/sentencepiece/issues/419
Refactored LLMTokenizer class methods to improve readability and maintainability.
Disabled initialization of NERDSTASH TOKENIZER as it causes errors on non-English hostnames or paths with spaces: https://github.com/google/sentencepiece/issues/419