Add more Tokenizer functionality - Create without download sync/async ; Trim APIs

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.

https://dot.net/ml

MIT License

9.01k stars 1.88k forks source link

Closed tarekgh closed 6 months ago

tarekgh commented 7 months ago

We need to incorporate the following enhancements into the tokenizer:

Enable the creation of tokenizers with streaming capability to avoid on-demand downloading of vocabulary files.
Introduce an API to facilitate encoding up to a specified maximum token count.
~~Introduce API to support encoding text from the end up to the maximum count.~~