`dotnet test --filter` appears to have a hidden character count limitation for tests with long names

DrEsteban commented 2 years ago

Describe the bug

It's a bit lengthy to describe, but dotnet test --filter seems to have a hidden max character length at which it will no longer match the filter string. Hopefully the repro steps below help clarify the issue.

NOTE: The filter string examples below might look a little weird, but according to the docs you essentially need to URL-escape special characters in order for them to be interpreted.

To Reproduce

Check-out my example repo: https://github.com/DrEsteban/dotnet-test-filter-bug-example
Open a command prompt to the repo location
Execute the following command, which works: dotnet test --filter "(DisplayName~TestProject1.UnitTest1.ThisIsAnExtremelyLongTestName_SoICanDemonstrateAStrangeBehaviorInDotnetTest_Filter&DisplayName~string1%3a %221234567890%22%2c string2%3a %221234567890%22%2c string3%3a %221234567890%22%2c string4%3a %221234567890%22%2c string5%3a %221234567890%22%2c string6%3a %221234567890%22%2c string7%3a %221234567890%22%2c string8%3a %221234567890%22%2c string9%3a %221234567890%22%2c string10%3a %221234567890%22%2c string11%3a %221234567890%22%2c string12%3a %221234567890%22%2c string13%3a %221234567890%22%2c string14%3a %22123456%22%2c string15%3a %221234)"
Execute the following command, which does not work: dotnet test --filter "(DisplayName~TestProject1.UnitTest1.ThisIsAnExtremelyLongTestName_SoICanDemonstrateAStrangeBehaviorInDotnetTest_Filter&DisplayName~string1%3a %221234567890%22%2c string2%3a %221234567890%22%2c string3%3a %221234567890%22%2c string4%3a %221234567890%22%2c string5%3a %221234567890%22%2c string6%3a %221234567890%22%2c string7%3a %221234567890%22%2c string8%3a %221234567890%22%2c string9%3a %221234567890%22%2c string10%3a %221234567890%22%2c string11%3a %221234567890%22%2c string12%3a %221234567890%22%2c string13%3a %221234567890%22%2c string14%3a %22123456%22%2c string15%3a %2212345)"

Note the only difference between those two filter strings is the single character 5 at the end. So, there seems to be a test name length at which the --filter parameter no longer works. This is further evidenced by:

Execute the following command, which does not work: dotnet test --filter "(DisplayName~TestProject1.UnitTest1.ThisIsAnExtremelyLongTestName_SoICanDemonstrateAStrangeBehaviorInDotnetTest_Filter&DisplayName~string16%3a %221234567890%22)"

This indicates that the character limit doesn't exist on the filter string, but rather on the test name itself.

E.g. TestProject1.UnitTest1.ThisIsAnExtremelyLongTestName_SoICanDemonstrateAStrangeBehaviorInDotnetTest_Filter(string1: "1234567890", string2: "1234567890", string3: "1234567890", string4: "1234567890", string5: "1234567890", string6: "1234567890", string7: "1234567890", string8: "1234567890", string9: "1234567890", string10: "1234567890", string11: "1234567890", string12: "1234567890", string13: "1234567890", string14: "123456", string15: "1234~567890", string16: "1234567890")~

^ None of the strikethrough portion above will match against the test name if used in the filter string.

Further technical details

Include the output of dotnet --info


.NET SDK (reflecting any global.json):
Version:   6.0.203
Commit:    a20feadf6d

Runtime Environment: OS Name: Windows OS Version: 10.0.22000 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.203\

global.json file: Not found

Host: Version: 6.0.9 Architecture: x64 Commit: 163a63591c

.NET SDKs installed: 1.1.14 [C:\Program Files\dotnet\sdk] 6.1.423 [C:\Program Files\dotnet\sdk] 7.0.202 [C:\Program Files\dotnet\sdk] 6.0.203 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.29 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 1.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 1.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.29 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.29 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Download .NET: https://aka.ms/dotnet-download

Learn about .NET Runtimes and SDKs: https://aka.ms/dotnet/runtimes-sdk-info

nohwnd commented 2 years ago

@engyebrahim please have a look on this to see if you can reproduce the issue on the latest net7 sdk.

engyebrahim commented 1 year ago

@nohwnd yes I still can reproduce it with net7 sdk.

nohwnd commented 1 year ago

Did you find what the issue is?

DrEsteban commented 1 year ago

@nohwnd I'm not sure who you're asking 😁, but I can still reproduce this issue with version 7.0.203 of the dotnet CLI. Personally I have no further insight, other than the continued belief it's something in the CLI/framework that's causing this behavior.

nohwnd commented 1 year ago

Sorry I was asking @engyebrahim but forgot to tag her.

dotnet / sdk