JUnit test failure is reported as success

[lefou]

Here is the output of a a minimal repro:

lefou:/tmp/example $ mill --disable-ticker __.test; echo "Exit code: $?"
Using mill version 0.9.5
Test run started
Running 3 tests in example.Test:
Test example.Test started
- test 1 - ok
Test example.Test finished, took 0.003 sec
Test example.Test started
- test 2 - fail *** FAILED ***
java.lang.AssertionError: Actual false is not true
        at de.tobiasroeser.lambdatest.Assert.fail(Assert.java:87)
        at de.tobiasroeser.lambdatest.Assert.assertTrue(Assert.java:320)
        at de.tobiasroeser.lambdatest.Expect.expectTrue(Expect.java:132)
        at de.tobiasroeser.lambdatest.Expect.expectTrue(Expect.java:145)
        at example.Test.lambda$new$1(Test.java:9)
        at de.tobiasroeser.lambdatest.junit.FreeSpecRunner.runChild(FreeSpecRunner.java:78)
        at de.tobiasroeser.lambdatest.junit.FreeSpecRunner.runChild(FreeSpecRunner.java:17)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.junit.runners.Suite.runChild(Suite.java:127)
        at org.junit.runners.Suite.runChild(Suite.java:26)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
        at com.novocode.junit.JUnitRunner$1.execute(JUnitRunner.java:132)
        at mill.scalalib.TestRunner$.$anonfun$runTests$2(TestRunner.scala:103)
        at scala.collection.immutable.ArraySeq.$anonfun$map$1(ArraySeq.scala:71)
        at scala.collection.immutable.ArraySeq.$anonfun$map$1$adapted(ArraySeq.scala:71)
        at scala.collection.immutable.ArraySeq$.tabulate(ArraySeq.scala:288)
        at scala.collection.immutable.ArraySeq$.tabulate(ArraySeq.scala:267)
        at scala.collection.ClassTagIterableFactory$AnyIterableDelegate.tabulate(Factory.scala:664)
        at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:71)
        at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:35)
        at mill.scalalib.TestRunner$.$anonfun$runTests$1(TestRunner.scala:82)
        at mill.modules.Jvm$.inprocess(Jvm.scala:193)
        at mill.scalalib.TestRunner$.runTests(TestRunner.scala:77)
        at mill.scalalib.TestRunner$.main(TestRunner.scala:53)
        at mill.scalalib.TestRunner.main(TestRunner.scala)
Test example.Test failed: Actual false is not true, took 0.012 sec
Test example.Test finished, took 0.012 sec
Test example.Test started
- test 3 - ok
Test example.Test finished, took 0.013 sec
Test run finished: 1 failed, 0 ignored, 3 total, 0.018s
Exit code: 0

This is the test class example/test/src/example/Test.java:

package example;

import de.tobiasroeser.lambdatest.junit.FreeSpec;
import static de.tobiasroeser.lambdatest.Expect.*;

public class Test extends FreeSpec {
  public Test() {
    test("test 1 - ok", () -> expectTrue(true));
    test("test 2 - fail", () -> expectTrue(false));
    test("test 3 - ok", () -> expectTrue(true));
  }
}

And the build file build.sc:

import mill._, scalalib._

object example extends JavaModule {
  object test extends Tests {
    def testFrameworks = Seq("com.novocode.junit.JUnitFramework")
    def ivyDeps = Agg(
      ivy"com.novocode:junit-interface:0.11",
      ivy"de.tototec:de.tobiasroeser.lambdatest:0.7.0",
      ivy"org.slf4j:slf4j-api:1.7.30"
    )
  }
}

Here's the outcome gathered by mill:

lefou:/tmp/example $ cat out/example/test/test/dest/out.json 
["",[{"fullyQualifiedName":"example.Test","selector":"example.Test","duration":3,"status":"Success"}]]

Looks like all three tests are mixed up. If I reduce the example to only include one failing test, the test failure is properly detected.

I already checked that the tests created by the JUnit Runner of FreeSpec / LambdaTest get unique names: https://github.com/lefou/LambdaTest/blob/0.7.0/src/main/java/de/tobiasroeser/lambdatest/junit/FreeSpecRunner.java#L45

[unkarjedy]

@lefou Looks like running JUnit tests from SBT produces correct reports if there are any failed tests. Are you sure that the issue is with junit-interface and not with Mill or LambdaTest? If you are sure, then what exactly is wrong with the reports?

[lefou]

The same example test suite is not detected to fail by sbt (which uses junit-interface). But it is properly detected to fail when run with Maven and Junit.

So, Junit properly runs the tests in all three build tools and also always detects one failure. But when junit-interface is used, the failure is not property reported. I think junit-interface is loosing some information, e.g. by ignoring some part of the test name.

Here are the build scripts I used:

sbt:

name := "Example"
libraryDependencies += "com.novocode" % "junit-interface" % "0.11"
libraryDependencies += "de.tototec" % "de.tobiasroeser.lambdatest" % "0.7.0"
libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.30"

Maven:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>org.example</groupId>
  <artifactId>jpademo</artifactId>
  <version>1.0</version>
  <packaging>jar</packaging>

  <properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>de.tototec</groupId>
      <artifactId>de.tobiasroeser.lambdatest</artifactId>
      <version>0.7.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>1.7.30</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

</project>

[lefou]

Fixed in release 0.12.