Tests defined in nested classes result in an error

[sanderploegsma]

At least two exercises in the Java track use nested classes in their unit tests using JUnit 4's @Enclosed annotation:

• REST API
• Simple Cipher

Running these exercises in the test runner results in the following errors:

REST API

```text $ bin/run-in-docker.sh rest-api ../exercism-track-java/exercises/practice/rest-api/ ./ Exception in thread "main" java.lang.NullPointerException: Null testCode at com.exercism.report.AutoValue_TestDetails$Builder.setTestCode(AutoValue_TestDetails.java:143) at com.exercism.report.ReportGenerator.buildTestDetails(ReportGenerator.java:25) at com.exercism.report.ReportGenerator.lambda$generate$0(ReportGenerator.java:13) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) at com.exercism.report.ReportGenerator.generate(ReportGenerator.java:13) at com.exercism.TestRunner.run(TestRunner.java:57) at com.exercism.TestRunner.main(TestRunner.java:40) ```

Simple Cipher

```text $ bin/run-in-docker.sh simple-cipher ../exercism-track-java/exercises/practice/simple-cipher/ ./ Exception in thread "main" java.lang.IllegalArgumentException: Multiple entries with same key: TestSource[packageName=, className=SimpleCipherTest, methodName=cipherCanEncode]=@Test public void cipherCanEncode() { String plainText = "aaaaaaaaaa"; String cipherText = "abcdefghij"; assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText); } and TestSource[packageName=, className=SimpleCipherTest, methodName=cipherCanEncode]=/** * Here we take advantage of the fact that plaintext of "aaa..." doesn't output the key. This is a critical * problem with shift ciphers, some characters will always output the key verbatim. */ @Test public void cipherCanEncode() { String plainText = "aaaaaaaaaa"; String cipherText = cipherWithDefaultKey.getKey().substring(0, 10); assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText); } at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:210) at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:204) at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:146) at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:109) at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:389) at com.exercism.junit.JUnitTestParser.buildTestCodeMap(JUnitTestParser.java:51) at com.exercism.TestRunner.run(TestRunner.java:57) at com.exercism.TestRunner.main(TestRunner.java:40) ```

The issue here is that the code that extracts the source from the test method isn't determining the class name correctly, causing the lookup to fail.

[sanderploegsma]

This probably also affects JUnit 5 test classes that use @Nested.

[sanderploegsma]

I tried fixing the logic that determines the class name for each test method and while that does resolve the error, it also yields interesting results:

Simple Cipher test results

```json { "status" : "pass", "tests" : [ { "name" : "keyIsLowercaseLetters", "test_code" : "@Test\npublic void keyIsLowercaseLetters() {\n assertThat(cipherWithDefaultKey.getKey()).matches(\"^[a-z]+$\");\n}", "status" : "pass" }, { "name" : "cipherCanDecode", "test_code" : "@Test\npublic void cipherCanDecode() {\n String cipherText = \"aaaaaaaaaa\";\n assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.getKey().substring(0, 10))).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanEncode", "test_code" : "/**\n * Here we take advantage of the fact that plaintext of \"aaa...\" doesn't output the key. This is a critical\n * problem with shift ciphers, some characters will always output the key verbatim.\n */\n@Test\npublic void cipherCanEncode() {\n String plainText = \"aaaaaaaaaa\";\n String cipherText = cipherWithDefaultKey.getKey().substring(0, 10);\n assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherIsReversible", "test_code" : "@Test\npublic void cipherIsReversible() {\n String plainText = \"abcdefghij\";\n assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.encode(plainText))).isEqualTo(plainText);\n}", "status" : "pass" }, { "name" : "cipherCanDoubleShiftEncode", "test_code" : "@Test\npublic void cipherCanDoubleShiftEncode() {\n String plainText = \"iamapandabear\";\n String cipherText = \"qayaeaagaciai\";\n assertThat(new Cipher(plainText).encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanDecode", "test_code" : "@Test\npublic void cipherCanDecode() {\n String plainText = \"abcdefghij\";\n String cipherText = \"aaaaaaaaaa\";\n assertThat(cipherWithDefaultKey.decode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanEncode", "test_code" : "@Test\npublic void cipherCanEncode() {\n String plainText = \"aaaaaaaaaa\";\n String cipherText = \"abcdefghij\";\n assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherMessageLongerThanKey", "test_code" : "@Test\npublic void cipherMessageLongerThanKey() {\n String plainText = \"iamapandabear\";\n String key = \"abc\";\n String cipherText = \"iboaqcnecbfcr\";\n assertThat(new Cipher(key).encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherIsReversibleGivenKey", "test_code" : "@Test\npublic void cipherIsReversibleGivenKey() {\n String plainText = \"abcdefghij\";\n assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.encode(plainText))).isEqualTo(plainText);\n}", "status" : "pass" }, { "name" : "cipherCanWrapDecode", "test_code" : "@Test\npublic void cipherCanWrapDecode() {\n String plainText = \"zabcdefghi\";\n String cipherText = \"zzzzzzzzzz\";\n assertThat(cipherWithDefaultKey.decode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanWrapEncode", "test_code" : "@Test\npublic void cipherCanWrapEncode() {\n String plainText = \"zzzzzzzzzz\";\n String cipherText = \"zabcdefghi\";\n assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanDoubleShiftEncode", "test_code" : "@Test\npublic void cipherCanDoubleShiftEncode() {\n String plainText = \"iamapandabear\";\n String cipherText = \"qayaeaagaciai\";\n assertThat(new Cipher(plainText).encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanDecode", "test_code" : "@Test\npublic void cipherCanDecode() {\n String plainText = \"abcdefghij\";\n String cipherText = \"aaaaaaaaaa\";\n assertThat(cipherWithDefaultKey.decode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanEncode", "test_code" : "@Test\npublic void cipherCanEncode() {\n String plainText = \"aaaaaaaaaa\";\n String cipherText = \"abcdefghij\";\n assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherMessageLongerThanKey", "test_code" : "@Test\npublic void cipherMessageLongerThanKey() {\n String plainText = \"iamapandabear\";\n String key = \"abc\";\n String cipherText = \"iboaqcnecbfcr\";\n assertThat(new Cipher(key).encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherIsReversibleGivenKey", "test_code" : "@Test\npublic void cipherIsReversibleGivenKey() {\n String plainText = \"abcdefghij\";\n assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.encode(plainText))).isEqualTo(plainText);\n}", "status" : "pass" }, { "name" : "cipherCanWrapDecode", "test_code" : "@Test\npublic void cipherCanWrapDecode() {\n String plainText = \"zabcdefghi\";\n String cipherText = \"zzzzzzzzzz\";\n assertThat(cipherWithDefaultKey.decode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanWrapEncode", "test_code" : "@Test\npublic void cipherCanWrapEncode() {\n String plainText = \"zzzzzzzzzz\";\n String cipherText = \"zabcdefghi\";\n assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "keyIsLowercaseLetters", "test_code" : "@Test\npublic void keyIsLowercaseLetters() {\n assertThat(cipherWithDefaultKey.getKey()).matches(\"^[a-z]+$\");\n}", "status" : "pass" }, { "name" : "cipherCanDecode", "test_code" : "@Test\npublic void cipherCanDecode() {\n String cipherText = \"aaaaaaaaaa\";\n assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.getKey().substring(0, 10))).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherCanEncode", "test_code" : "/**\n * Here we take advantage of the fact that plaintext of \"aaa...\" doesn't output the key. This is a critical\n * problem with shift ciphers, some characters will always output the key verbatim.\n */\n@Test\npublic void cipherCanEncode() {\n String plainText = \"aaaaaaaaaa\";\n String cipherText = cipherWithDefaultKey.getKey().substring(0, 10);\n assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}", "status" : "pass" }, { "name" : "cipherIsReversible", "test_code" : "@Test\npublic void cipherIsReversible() {\n String plainText = \"abcdefghij\";\n assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.encode(plainText))).isEqualTo(plainText);\n}", "status" : "pass" } ], "version" : 3 } ```

The problem here is that for some reason all tests are reported twice. This seems to be caused by JUnit discovering the tests multiple times, but I'm not sure.

[danilopiazza]

The problem here is that for some reason all tests are reported twice. This seems to be caused by JUnit discovering the tests multiple times, but I'm not sure.

I think you're right. I was able to reproduce this using the JUnit Platform Launcher API: https://github.com/junit-team/junit5/issues/3537.

This only happens on JUnit Vintage. JUnit 5 @Nested tests are not affected.

[sanderploegsma]

Thanks for sorting this out! Good to hear that JUnit 5 is not affected by this.

[sanderploegsma]

This issue should now be fixed!