Closed lapp0 closed 1 week ago
Fixes https://github.com/outlines-dev/outlines/issues/982
models.mlxlm didn't properly apply whitespace prefixes for some tokenizers such as phi3.
models.mlxlm
phi3
Use official mlx-lm implementations tokenizer.detokenizer to manage last-token str representation when iterating over token strings.
mlx-lm
tokenizer.detokenizer
Fixes https://github.com/outlines-dev/outlines/issues/982
Problem
models.mlxlmdidn't properly apply whitespace prefixes for some tokenizers such asphi3.Solution
Use official
mlx-lmimplementationstokenizer.detokenizerto manage last-token str representation when iterating over token strings.