Open yifanmai opened 3 months ago
Thanks @yifanmai for the contribution!
We intentionally skipped this when we introduced offline local tokenization; since we were optimizing for time. However, let me check how much overhead this adds (for bigger inputs) and if it's negligible we can add it.
Fixes #493