Microsoft.ML.Tokenizers.Tests.TiktokenTests.TestTokenizerUsingExternalVocab failing to download gpt2.tiktoken

[ericstj]

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=823668

Build leg reported

Microsoft.ML.Tokenizers.Tests.TiktokenTests.TestTokenizerUsingExternalVocab

Pull Request

https://github.com/dotnet/machinelearning/pull/7252

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : ["Microsoft.ML.Tokenizers.Tests.TiktokenTests.TestTokenizerUsingExternalVocab", "File not found"],
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• [ ] Feature changes/additions
• [ ] Bug fixes
• [x] Internal Infrastructure Improvements

  Release Note Description

Additional information about the issue reported

It looks to me like this file no longer exists: https://github.com/dotnet/machinelearning/blob/be1e428d41b5936903172855f7f30861ca7eb49a/test/Microsoft.ML.Tokenizers.Tests/TitokenTests.cs#L100C136-L100C149

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=823668 Error message validated: [Microsoft.ML.Tokenizers.Tests.TiktokenTests.TestTokenizerUsingExternalVocab File not found] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 10/4/2024 12:37:23 AM UTC

Report

Build	Definition	Test	Pull Request
823668	dotnet/machinelearning	Microsoft.ML.Tokenizers.Tests.WorkItemExecution	dotnet/machinelearning#7252

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	1

[dougbu]

@ericstj may I ask why this is a dotnet/dnceng issue❓ shouldn't it be moved into dontet/machinelearning❓

[tarekgh]

I think it is fine to move it to the machinelearning repo. I don't have permission to do so though.

[dougbu]

I don't have the required rights in machinelearning either. @mmitche can you transfer the issue❓

[ericstj]

Oh that's odd. Maybe I clicked the wrong button when opening this. I'll transfer. Sorry for that.

[tarekgh]

Should be fixed now by the PR https://github.com/dotnet/machinelearning/pull/7248.