microsoft / testfx

MSTest framework and adapter
MIT License
748 stars 255 forks source link

Provide additional information for assert failures #334

Open codito opened 6 years ago

codito commented 6 years ago

Description

Default Assert methods should provide information in case of failures. They should at least answer the next question for a developer once a failure occurs. I've seen such behavior with CollectionAsserts, request to review other APIs as well.

Steps to reproduce

  1. Create a hello world test project
  2. Add two List<int> variables. var actual = { 1, 2, 3 } and expected = { 1, 3, 2 }
  3. Add an assert. CollectionAssert.AreEqual(expected, actual)
  4. Run tests

Expected behavior

A developer would immediately ask what were the elements if the collection doesn't match. Note the **additional** information below.

Failed   C.Tests.N.D
Error Message:
 CollectionAssert.AreEqual failed. (Element at index 1 do not match.)
 **Expected = { 2, 1, 3 }, Actual = { 1, 2, 3 }**
Stack Trace:
   at C.Tests.N.D()

Actual behavior

Failed   C.Tests.N.D
Error Message:
 CollectionAssert.AreEqual failed. (Element at index 1 do not match.)
Stack Trace:
   at C.Tests.N.D()

Environment

MSTest 1.2.0, VSTest 15.3.0, dotnet-sdk 2.0 (linux)

AB#2112230

GoFightNguyen commented 6 years ago

This is a great issue. If this functionality gets added, then great. But I think we should also consider using Fluent Assertions.

I used your example and Fluent Assertions for the asserts.

actual.Should().BeEquivalentTo(expected); This passes, because it does not care about ordering.

If I do actual.Should().BeEquivalentTo(expected, o => o.WithStrictOrdering());, then the test output is

Message: Expected item[1] to be a collection with 3 item(s)Expected item[1] to be 3, but found 2. Expected item[2] to be a collection with 3 item(s)Expected item[2] to be 2, but found 3.

Admittedly, that message has unnecessary and redundant information

Expected item[1] to be a collection with 3 item(s) Expected item[2] to be a collection with 3 item(s)

I'll have to open an issue, because the previous version of Fluent Assertions did not include that cruft.