search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.05k stars 1.88k forks source link

Misc Changes #7264

Closed tarekgh closed 1 month ago

tarekgh commented 1 month ago

The changes here include the following:

Every commit in the PR is representing one of the listed changes. Reviewing every commit separately will make it easier to have a clear understanding of the changes.

tarekgh commented 1 month ago

@michaelgsharp Could you please help review this change?

@ericstj could you please help reviewing the changes in the csproj files for a new added Microsoft.Bcl.Memory dependency and enable restoring from net9 feed? The second commit include such changes.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 92.19858% with 22 lines in your changes missing coverage. Please review.

Project coverage is 68.80%. Comparing base (9baf26b) to head (af53b84). Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs 84.21% 5 Missing and 1 partial :warning:
src/Microsoft.ML.TorchSharp/Loss/FocalLoss.cs 80.00% 4 Missing :warning:
...crosoft.ML.GenAI.Core/Pipeline/CausalLMPipeline.cs 0.00% 3 Missing :warning:
...oft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs 66.66% 2 Missing and 1 partial :warning:
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs 90.00% 0 Missing and 2 partials :warning:
src/Microsoft.ML.TorchSharp/Utils/RangeUtil.cs 60.00% 0 Missing and 2 partials :warning:
src/Microsoft.ML.Tokenizers/Model/Phi2Tokenizer.cs 0.00% 1 Missing :warning:
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs 90.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #7264 +/- ## ========================================== + Coverage 68.78% 68.80% +0.01% ========================================== Files 1463 1461 -2 Lines 272288 272405 +117 Branches 28177 28176 -1 ========================================== + Hits 187299 187427 +128 + Misses 77745 77743 -2 + Partials 7244 7235 -9 ``` | [Flag](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | Coverage Δ | | |---|---|---| | [Debug](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `68.80% <92.19%> (+0.01%)` | :arrow_up: | | [production](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `63.29% <86.66%> (+<0.01%)` | :arrow_up: | | [test](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | `89.07% <100.00%> (+0.02%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files with missing lines](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | Coverage Δ | | |---|---|---| | [...c/Microsoft.ML.GenAI.LLaMA/LlamaTokenizerHelper.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.GenAI.LLaMA%2FLlamaTokenizerHelper.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5HZW5BSS5MTGFNQS9MbGFtYVRva2VuaXplckhlbHBlci5jcw==) | `100.00% <100.00%> (ø)` | | | [src/Microsoft.ML.Tokenizers/EncodedToken.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.Tokenizers%2FEncodedToken.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL0VuY29kZWRUb2tlbi5jcw==) | `100.00% <100.00%> (ø)` | | | [...rc/Microsoft.ML.Tokenizers/Model/LlamaTokenizer.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.Tokenizers%2FModel%2FLlamaTokenizer.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL01vZGVsL0xsYW1hVG9rZW5pemVyLmNz) | `59.09% <ø> (ø)` | | | [...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.Tokenizers%2FModel%2FTiktokenTokenizer.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL01vZGVsL1Rpa3Rva2VuVG9rZW5pemVyLmNz) | `78.28% <100.00%> (+0.37%)` | :arrow_up: | | [src/Microsoft.ML.Tokenizers/Model/Word.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.Tokenizers%2FModel%2FWord.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL01vZGVsL1dvcmQuY3M=) | `61.42% <100.00%> (ø)` | | | [...crosoft.ML.Tokenizers/PreTokenizer/PreTokenizer.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.Tokenizers%2FPreTokenizer%2FPreTokenizer.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL1ByZVRva2VuaXplci9QcmVUb2tlbml6ZXIuY3M=) | `100.00% <100.00%> (ø)` | | | [...ft.ML.Tokenizers/PreTokenizer/RegexPreTokenizer.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.Tokenizers%2FPreTokenizer%2FRegexPreTokenizer.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL1ByZVRva2VuaXplci9SZWdleFByZVRva2VuaXplci5jcw==) | `87.23% <100.00%> (ø)` | | | [src/Microsoft.ML.Tokenizers/Tokenizer.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.Tokenizers%2FTokenizer.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub2tlbml6ZXJzL1Rva2VuaXplci5jcw==) | `84.57% <100.00%> (ø)` | | | [...rc/Microsoft.ML.TorchSharp/AutoFormerV2/Anchors.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.TorchSharp%2FAutoFormerV2%2FAnchors.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub3JjaFNoYXJwL0F1dG9Gb3JtZXJWMi9BbmNob3JzLmNz) | `88.13% <100.00%> (ø)` | | | [.../Microsoft.ML.TorchSharp/AutoFormerV2/Attention.cs](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree&filepath=src%2FMicrosoft.ML.TorchSharp%2FAutoFormerV2%2FAttention.cs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet#diff-c3JjL01pY3Jvc29mdC5NTC5Ub3JjaFNoYXJwL0F1dG9Gb3JtZXJWMi9BdHRlbnRpb24uY3M=) | `98.87% <100.00%> (ø)` | | | ... and [18 more](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet) | | ... and [10 files with indirect coverage changes](https://app.codecov.io/gh/dotnet/machinelearning/pull/7264/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dotnet)