search

nunit / nunit-vs-adapter

Runs NUnit V2 tests inside the Visual Studio 2012 or later Test Explorer window.
MIT License
50 stars 43 forks source link

Weird `TestCase` attribute behaviour when using specific character combinations #173

Closed ghost closed 5 years ago

ghost commented 5 years ago

I have a bit of an odd failure at hand here and I can't figure out what exactly is going on.

The Setup

I have a .NET Standard 2.0 test project that looks similar to this:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net461;netcoreapp2.0</TargetFrameworks>
    <Configurations>Debug;Release</Configurations>
  </PropertyGroup>

[... various shared project references in here that contain the tests ... ]

  <ItemGroup>
    <PackageReference Include="Microsoft.CSharp" Version="4.4.1" />
    <PackageReference Include="NLog" Version="4.5.3" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.5.0" />
    <PackageReference Include="NSubstitute" Version="3.1.0" />
    <PackageReference Include="NUnit" Version="3.9.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.9.0" />
  </ItemGroup>
</Project>

The tests get executed via dotnet test [name].csproj -c Release -v n -f [netcoreapp2.0/net461] on various machines and operating systems. The tests run all fine on a wide range of machines and operating systems however there is a single machine out there that is showing us an interesting failure.

The Test Failure

The test in question has a TestCase attribute with a single string argument: [TestCase( "{\"\":{\"\ud800\udd51\":\"team\"}}}}" )]. The meaning of the string is not relevant for this test and as you can see it contains a Unicode character that is made up of two escape sequences.

This test case works fine on Windows, OSX, and Ubuntu machines with the exception of a single Ubuntu machine that seems to be transforming the TestCase string argument into something invalid. \ud800\udd51 shows up as \0\udd51 on this particular machine and this is where the head scratching begins.

I wrote a very simple test case to isolate the issue as best as I could and this is how far I got:

[TestCase( "{\"\":{\"\ud800\udd51\":\"team\"}}}}" )] // A
[TestCase( "{\"\":{\"\ud800\udd51\":" )] // B
[TestCase( "\u6c34\ud800\udd51" )] // C
[TestCase( "\"\ud800\udd51\"" )] // D
public void StringTest( string s) {
    var j = System.Text.Encoding.Unicode.GetBytes(s);
    Assert.AreEqual( s, System.Text.Encoding.Unicode.GetString(j) );
}

[Test] // E
public void StringTest2() {
    StringTest( "{\"\":{\"\ud800\udd51\":\"team\"}}}}" );
}

Test cases B, C, D, and E all pass and show the correct unicode character in the console summary while test case A fails and shows the following:

"{\"\":{\"\0\udd51\":\"team\"}}}}"
---
Expected: "{"":{"\0\udd51":"team"}}}}"
But was:  "{"":{"\0�":"team"}}}}"

For some reason test case A transforms the high surrogate into a \0 and so the second block results in an invalid unicode character (\udd51 is invalid). The question here is why and how is this happening? Especially as other tests that use \ud800\udd51 pass just fine.

The Machine

As far as I can tell there is nothing out of the ordinary about the machine that is trying to run the tests. Here is the dotnet --info output:

.NET Core SDK (reflecting any global.json):
 Version:   2.1.302
 Commit:    9048955601

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.302/

Host (useful for support):
  Version: 2.1.2
  Commit:  811c3ce6c0

.NET Core SDKs installed:
  2.1.302 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

To make matters worse other similar Ubuntu machines run the tests just fine. I had no luck reproducing this issue on any other machine so far...

ghost commented 5 years ago

Update:

It turns out other Ubuntu machines without the failure had old .NET Core versions installed as well:

.NET Core SDK (reflecting any global.json):
 Version:   2.1.302
 Commit:    9048955601
Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.302/
Host (useful for support):
  Version: 2.1.2
  Commit:  811c3ce6c0
.NET Core SDKs installed:
  2.1.200 [/usr/share/dotnet/sdk]
  2.1.300-rc1-008673 [/usr/share/dotnet/sdk]
  2.1.302 [/usr/share/dotnet/sdk]
.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0-rc1-final [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0-rc1-final [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0-rc1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

After removing these and only keeping the 2.1.302 relevant bits the tests seem to fail on the other machines now. I am currently setting up a clean environment to see if I get the same results and then try to dig deeper.

ghost commented 5 years ago

A few more observations to this issue: I got hold of a fresh Ubuntu install and installed the latest .NET Core. I now can reliably reproduce the error. I tried upgrading various components to the newest version one by one to see if it would fix the issue but it is still present. As of now the project uses the following dependencies:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.8.0" />
<PackageReference Include="NUnit" Version="3.10.1" />
<PackageReference Include="NUnit3TestAdapter" Version="3.10.0" />

I also tried to change the target framework to netcoreapp2.1 but this didn't help either. The error persists.

Something odd about the test case string:

So it seems to me that it has something to do with the length of the string that I pass in. Specifically if \ud800\udd51 is within a string of >= 14 characters.

ghost commented 5 years ago

Whoops. I just realized this belongs to the NUnit3 test adapter and not the V2 one. Sorry about this. The new issue can be found here: https://github.com/nunit/nunit3-vs-adapter/issues/534