Closed maettu-this closed 4 months ago
This isn't a TestCentric issue - the test fails and TestCentric reports it accordingly.
If you report it to the NUnit team, I'm sure they'll tell you it's not an issue there either but by design.
In NUnit V2, it was possible for a test to catch it's own failing assertion and change the result so that the failing test passed. That is no longer possible with NUnit 3. An exception is still used in order to cause the failing test to to exit but the failure is reported and recorded before that ever happens. The general philosophy was that a test should not be able to "cancel" its own failure.
There are different ways to handle this depending on the actual use case but catching the exception is no longer a useful option.
@CharliePoole, thanks for the background. Knowning this, I have found https://github.com/nunit/nunit/issues/2758 dealing with this topic, including the hint on using IsolatedContext()
.
I guess there were good reasons for changing this behavior. I disagree with @rprouse's comment don't see that as a breaking change, for two reasons:
AssertionException
is thrown. And Assert
leading to an AssertionException
seems obvious logic.It's obviously hopeless to try convincing the NUnit team to bring back this behavior. I will have to adapt my test implementations where I catch AssertionException
now. Uff, another x hours this upgrade is taking...
PS: I catch AssertionException
for dealing with instabilities in the test setup, instabilities in communication with physical devices. In case such test fails, it evaluates the reason and error tolerance and then either cancels or rethrows. For handling this, neither Retry
nor IsolatedContext
sounds like the right thing to do. And Assert.Throws
even less, as the test fails only in e.g. 1 out of a thousand times. But I'm sure I'll find a way to handle this.
I'd say you're right about trying to bring it back. Your arguments (made by other people) were considered back at the time the change was made.
Regarding "breaking" I now believe that every project needs a written definition of what they mean by breaking. Otherwise, users push for the loosest possible meaning. :-)
Giving an update on this topic, the discussion in https://github.com/nunit/nunit/issues/4701 has actually led to another approach to deal with this issue:
NUnit.Framework.Internal.TestExecutionContext.CurrentContext.CurrentResult.AssertionResults.Clear();
NUnit might even provide a non-internal wrapper method for this approach in the future.
Thanks, I added a comment on that issue. Frankly, as noted, it still seems like a bad idea to me.
Describe the bug The attached repro sample implements a test where an assertion fails but its exception is handled, i.e. the test is expected to be indicated
Passed
. But the TestCentric GUI reports the test asFailed
.To Reproduce Steps to reproduce the behavior:
Debug
orRelease
, doesn't matter).SomeTest
.Expected behavior The test's result is indicated
Passed
.Screenshots![grafik](https://github.com/TestCentric/testcentric-gui/assets/24897735/42d910c5-4fe7-4a2e-8c47-c2284c0a2e44)
Environment (please complete the following information):