search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.05k stars 1.88k forks source link

Move the Tokenizer's data into separate packages. #7248

Closed tarekgh closed 1 month ago

tarekgh commented 1 month ago

CC @ericstj @stephentoub @luisquintanilla @michaelgsharp

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.

Project coverage is 68.78%. Comparing base (be1e428) to head (9df82d6). Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
test/Microsoft.ML.Tokenizers.Tests/Utils.cs 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #7248 +/- ## ======================================= Coverage 68.77% 68.78% ======================================= Files 1462 1463 +1 Lines 272261 272288 +27 Branches 28176 28177 +1 ======================================= + Hits 187254 187286 +32 + Misses 77764 77759 -5 Partials 7243 7243 ``` | [Flag](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | Coverage Δ | | |---|---|---| | [Debug](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `68.78% <97.56%> (+<0.01%)` | :arrow_up: | | [production](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `63.28% <100.00%> (+<0.01%)` | :arrow_up: | | [test](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `89.04% <96.29%> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files with missing lines](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | Coverage Δ | | |---|---|---| | [...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248?src=pr&el=tree&filepath=src%2FMicrosoft.ML.Tokenizers%2FModel%2FTiktokenTokenizer.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL01vZGVsL1Rpa3Rva2VuVG9rZW5pemVyLmNz) | `77.91% <100.00%> (+0.08%)` | :arrow_up: | | [...oft.ML.Tokenizers.Data.Tests/TokenizerDataTests.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248?src=pr&el=tree&filepath=test%2FMicrosoft.ML.Tokenizers.Data.Tests%2FTokenizerDataTests.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-dGVzdC9NaWNyb3NvZnQuTUwuVG9rZW5pemVycy5EYXRhLlRlc3RzL1Rva2VuaXplckRhdGFUZXN0cy5jcw==) | `100.00% <100.00%> (ø)` | | | [...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248?src=pr&el=tree&filepath=test%2FMicrosoft.ML.Tokenizers.Tests%2FTiktokenTests.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-dGVzdC9NaWNyb3NvZnQuTUwuVG9rZW5pemVycy5UZXN0cy9UaWt0b2tlblRlc3RzLmNz) | `98.98% <100.00%> (ø)` | | | [test/Microsoft.ML.Tokenizers.Tests/Utils.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248?src=pr&el=tree&filepath=test%2FMicrosoft.ML.Tokenizers.Tests%2FUtils.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-dGVzdC9NaWNyb3NvZnQuTUwuVG9rZW5pemVycy5UZXN0cy9VdGlscy5jcw==) | `70.00% <0.00%> (ø)` | | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/dotnet/machinelearning/pull/7248/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet)
tarekgh commented 1 month ago

CC @LittleLittleCloud for awareness.