This issue to track investigating and address the feedback we got regarding the tokenizers design
@stephentoub Feedback
If we’re able to make such breaking changes, we should also be reconsidering other aspects of the library then I think, in particular for perf, for example:
[x] Token is class, which means allocation per token, plus the design effectively forces the string Value of the Token to be materialized even if it’s never used.
[x] I don’t see any way to get just a token count without materializing the list of tokens, even though just the count is a commonly needed thing in these scenarios. Presumably such an API could get away with a lot less overhead / allocation. https://github.com/dotnet/machinelearning/pull/7024
This issue to track investigating and address the feedback we got regarding the tokenizers design
@stephentoub Feedback
If we’re able to make such breaking changes, we should also be reconsidering other aspects of the library then I think, in particular for perf, for example:
[x] Token is class, which means allocation per token, plus the design effectively forces the string Value of the Token to be materialized even if it’s never used.
[x] I don’t see any way to get just a token count without materializing the list of tokens, even though just the count is a commonly needed thing in these scenarios. Presumably such an API could get away with a lot less overhead / allocation. https://github.com/dotnet/machinelearning/pull/7024
[x] Should there be support for spans baked in? https://github.com/dotnet/machinelearning/pull/7035
[x] https://github.com/dotnet/machinelearning/pull/7024