Multiple failed ErrorCollector assertions in parameterized tests causes MultipleFailureException resulting in a test error

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Create a new test class with annotation "@RunWith(JUnitParamsRunner.class)"
2. Create a new ErrorCollector instance variable for the tests:
@Rule public ErrorCollector collector = new ErrorCollector();
3. Create a new method that provides the parameters, e.g.:
private Object[] provideParams() { return $($(false, true)); }
4. Create a new test method that uses the parameters and fails multiple 
assertions, e.g.:
@Test @Parameters(method="provideParams")
public void testMultipleFailedAssertionsWithParams(boolean first, boolean 
second) {
   collector.checkThat(first, is(second));
   collector.checkThat(first, is(second));
}
5. Run Test.

What is the expected output? What do you see instead?
* The expected result of the test is that it fails with multiple 
junit.framework.AssertionFailedError instances and prints the corresponding 
stack traces to the console, so that locating the failing statements is 
possible.
* However, instead the test yields an error. The 
junit.framework.AssertionFailedError instances get only partially printed to 
the console. The stack traces that show at which statements the test cases have 
failed are non-existant. Instead the following stack trace appears in the 
console:

org.junit.internal.runners.model.MultipleFailureException: There were 2 errors:
    at junitparams.internal.ParameterisedTestMethodRunner.runMethodInvoker(ParameterisedTestMethodRunner.java:48)
    at junitparams.internal.ParameterisedTestMethodRunner.runTestMethod(ParameterisedTestMethodRunner.java:42)
    at junitparams.internal.ParameterisedTestClassRunner.runParameterisedTest(ParameterisedTestClassRunner.java:143)
    at junitparams.JUnitParamsRunner.runChild(JUnitParamsRunner.java:405)
    at junitparams.JUnitParamsRunner.runChild(JUnitParamsRunner.java:383)

What version of the product are you using? On what operating system?
* JUnitParams 1.0.2
* JUnit 4.11
* Hamcrest Core 1.3
* Windows 7
* Netbeans 7.4 & 8.0

Please provide any additional information below.
I've attached an example implementation that implements 4 test cases that use 
the error collector:
* 2 are not paremeterized:
  - one of them causes 1 failure <- expected result. (1 Failure, correct stacktrace)
  - one of them causes 2 failures <- expected result. (2 Failures, correct stacktraces)
* 2 are parameterized:
  - one of them causes 1 failure <- expected result. (1 Failure, correct stacktrace)
  - one of them causes 2 failures <- unexpected result. (2 Errors, org.junit.internal.runners.model.MultipleFailureException)

I've also attached a console log of the execution of the provided test class:
* Line 04-27 shows the expected failures.
* Line 30-49 shows the error.
* Line 52-79 shows the expected failures.

Best regards,
Philipp

Original issue reported on code.google.com by phil.gra...@gmail.com on 21 Feb 2014 at 2:28

Attachments:

• MultipleFailedAssertionsTest.java
• test.log

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Forgot to add: In the example implementation the test cases are:
* testSingleFailedAssertion (not-parameterized) causes 1 failure <- expected 
result. (correct stacktrace)
* testMultipleFailedAssertions (not-parameterized) causes 2 failures <- 
expected result. (correct stacktraces)
* testSingleFailedAssertionWithParams (parameterized) causes 1 failure <- 
expected result. (correct stacktrace)
* testMultipleFailedAssertionsWithParams (parameterized) causes 2 errors <- 
unexpected result. (org.junit.internal.runners.model.MultipleFailureException)

Best regards,
Philipp

Original comment by phil.gra...@gmail.com on 21 Feb 2014 at 2:35

[GoogleCodeExporter]

Original comment by lipinski...@gmail.com on 14 Apr 2014 at 2:37

• Changed state: Accepted

[GoogleCodeExporter]

Original comment by lipinski...@gmail.com on 15 Apr 2014 at 12:11

• Changed state: Done