dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
8.91k stars 1.86k forks source link

Add Tiktoken Synchronous Creation Using Model Name #7080

Closed tarekgh closed 3 months ago

tarekgh commented 3 months ago

Fixes https://github.com/dotnet/machinelearning/issues/7077

The change:

tarekgh commented 3 months ago

CC @tonybaloney @ericstj @michaelgsharp

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 82.25806% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 68.82%. Comparing base (8b483f4) to head (3038075).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #7080 +/- ## ======================================= Coverage 68.82% 68.82% ======================================= Files 1255 1255 Lines 250358 250400 +42 Branches 25550 25554 +4 ======================================= + Hits 172310 172344 +34 - Misses 71438 71443 +5 - Partials 6610 6613 +3 ``` | [Flag](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | Coverage Δ | | |---|---|---| | [Debug](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `68.82% <82.25%> (+<0.01%)` | :arrow_up: | | [production](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `63.26% <68.57%> (+<0.01%)` | :arrow_up: | | [test](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `88.51% <100.00%> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | Coverage Δ | | |---|---|---| | [...crosoft.ML.Tokenizers/Utils/Helpers.netstandard.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL1V0aWxzL0hlbHBlcnMubmV0c3RhbmRhcmQuY3M=) | `81.81% <100.00%> (+0.56%)` | :arrow_up: | | [test/Microsoft.ML.Tokenizers.Tests/TitokenTests.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-dGVzdC9NaWNyb3NvZnQuTUwuVG9rZW5pemVycy5UZXN0cy9UaXRva2VuVGVzdHMuY3M=) | `100.00% <100.00%> (ø)` | | | [src/Microsoft.ML.Tokenizers/Model/Tiktoken.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL01vZGVsL1Rpa3Rva2VuLmNz) | `68.55% <67.64%> (+0.83%)` | :arrow_up: | ... and [5 files with indirect coverage changes](https://app.codecov.io/gh/dotnet/machinelearning/pull/7080/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet)
tarekgh commented 3 months ago

I filed https://github.com/dotnet/machinelearning/issues/7084 for the failures but build analysis didn't enable to squash and merge.