Closed tarekgh closed 3 months ago
CC @tonybaloney @ericstj @michaelgsharp
Attention: Patch coverage is 82.25806%
with 11 lines
in your changes are missing coverage. Please review.
Project coverage is 68.82%. Comparing base (
8b483f4
) to head (3038075
).
I filed https://github.com/dotnet/machinelearning/issues/7084 for the failures but build analysis didn't enable to squash and merge.
Fixes https://github.com/dotnet/machinelearning/issues/7077
The change:
Tiktoken.CreateTokenizerByModelName
allow synchronous creation of the tokenizer using the model name.Tiktoken.CreateByModelNameAsync
toTiktoken.CreateTokenizerByModelNameAsync
. This adjustment is being made because theTiktoken
class includes other methods for creating models, not just tokenizers. By adding the wordTokenizer
to the method responsible for creating a tokenizer, it will provide clarity regarding its functionality.