Theory with ClassData does not fail as expected

[thomasgalliker]

I just created a test [Theory] with [ClassData] as input. When one of the test case fails, xunit.devices still shows the whole test as successful. I tried it on WP8 test runner only. If it is the last Theory that fails, the whole test is failed. Can you reproduce this error?

    [Theory]
    [ClassData(typeof(PropertyInputData))]
    public void ShouldReadWriteNativeTypes(ITestValue testValue)
    {
        // Arrange
        var inputValue = testValue.Value;

        // Act
        var outputValue = ReadWriteValueOfType(inputValue);

        // Assert // HERE you can manipulate the Assertion to reproduce the error
        Assert.Equal(inputValue, outputValue);
    }

    public class PropertyInputData : IEnumerable<object[]>
    {
        private readonly List<object[]> data = new List<object[]>
        {
            new object[] { new TestValue<bool>(true) },
            new object[] { new TestValue<short>(short.MaxValue) },
            new object[] { new TestValue<ushort>(ushort.MaxValue) },
            new object[] { new TestValue<int>(int.MaxValue) },
            new object[] { new TestValue<uint>(uint.MaxValue) },
            new object[] { new TestValue<float>(float.MaxValue) },
        };

        public IEnumerator<object[]> GetEnumerator()
        {
            return this.data.GetEnumerator();
        }

        IEnumerator IEnumerable.GetEnumerator()
        {
            return this.GetEnumerator();
        }
    }

[thomasgalliker]

After some hours of investigation, I found out that xunit does not list theories as seperate testcases as long as the parameter types are not serializable. Found this on stackoverflow:

"Make sure every object (and any non-primitive it may contain) can be fully serialized and deserialized. .NET's [Serializable] attribute (and ISerializable if you need more control), or xUnit's IXunitSerializable both work to tell xUnit that it's a serializable object."

http://stackoverflow.com/questions/30574322/xunit-memberdata-tests-show-up-as-one-test-instead-of-many

[Mardoxx]

Pretty sure that's wrong. You have to implement IXunitSerializable for it to work. https://github.com/xunit/xunit/blob/master/src/common/XunitSerializationInfo.cs#L104

Something like this would work, but you'd have to ask why you want to do it this way...

public class TestClient : IXunitSerializable
{
    public TestClient()
    {

    }

    public TestClient(Client client)
    {
        Client = client;
    }

    public Client Client { get; protected set; }

    public void Deserialize(IXunitSerializationInfo info)
    {
        var json = info.GetValue<string>("json");
        Client = JsonConvert.DeserializeObject<Client>(json);
    }

    public void Serialize(IXunitSerializationInfo info)
    {
        var json = JsonConvert.SerializeObject(Client);
        info.AddValue("json", json);
    }
}

public static IEnumerable<object[]> TestClients()
{
    yield return new object[] { new TestClient(new Client { Enabled = false }) }; // Should fail
    yield return new object[] { new TestClient(new Client { Enabled = true }) };
}

[Theory]
[MemberData(nameof(TestClients))]
public void TestClientsTests(TestClient testClient)
{
    var client = testClient.Client;

    Assert.True(client.Enabled);
}