0.12.2, by "gradle test" 2 tests failed, but each test passes when executed one by one

[kazurayam]

0.12.2 when I executed "gradle test", 2 tests failed. But when I execute each, both passed.

1. com.kazurayam.materialstore.base.reduce.MaterialProductGroup_issue87Test
2. issues.Issue73Test

output of MaterialProductGroup_issue87Test

org.opentest4j.AssertionFailedError: expected: <8> but was: <12>
    at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
    at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
    at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
    at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
    at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
    at com.kazurayam.materialstore.base.reduce.MaterialProductGroup_issue87Test.test_getQueryOnMetadataList(MaterialProductGroup_issue87Test.java:109)
...

Stdout (This output is very suspicious)

[test_getQueryOnMetadataList]
queryList.get(0): {"URL.query":"30af91bf14e37666a085fb8a161ff36d"}
queryList.get(1): {}
queryList.get(2): {}
queryList.get(3): {}
queryList.get(4): {}
queryList.get(5): {}
queryList.get(6): {}
queryList.get(7): {}
queryList.get(8): {}
queryList.get(9): {}
queryList.get(10): {}
queryList.get(11): {}

Stderr

[Test worker] INFO com.kazurayam.materialstore.base.reduce.zipper.Zipper - [zipMaterials] foundLeftCount=2 is unusual
[Test worker] INFO com.kazurayam.materialstore.base.reduce.zipper.Zipper - [zipMaterials] foundLeftCount=2 is unusual
[Test worker] INFO com.kazurayam.materialstore.base.reduce.zipper.Zipper - [zipMaterials] foundLeftCount=2 is unusual
...

[kazurayam]

When I execute Issue73Test independently, I got the following message

[test_getQueryOnMetadataList]
queryList.get(0): {"URL.path":"/", "URL.port":"80", "URL.protocol":"http"}
queryList.get(1): {"URL.path":"/", "URL.port":"80", "URL.protocol":"http"}
queryList.get(2): {"URL.path":"/ajax/libs/jquery/1.12.4/jquery.js", "URL.port":"80", "URL.protocol":"https"}
queryList.get(3): {"URL.path":"/npm/bootstrap-icons@1.7.2/font/bootstrap-icons.css", "URL.port":"80", "URL.protocol":"https"}
queryList.get(4): {"URL.path":"/npm/bootstrap-icons@1.7.2/font/fonts/bootstrap-icons.woff2", "URL.port":"80", "URL.protocol":"https", "URL.query":"30af91bf14e37666a085fb8a161ff36d"}
queryList.get(5): {"URL.path":"/npm/bootstrap@5.1.0/dist/css/bootstrap.min.css", "URL.port":"80", "URL.protocol":"https"}
queryList.get(6): {"URL.path":"/npm/bootstrap@5.1.0/dist/js/bootstrap.bundle.min.js", "URL.port":"80", "URL.protocol":"https"}
queryList.get(7): {"URL.path":"/umineko-1960x1960.jpg", "URL.port":"80", "URL.protocol":"http"}

This is OK. Things should go like this always.

[kazurayam]

When I executed the issue87Test only, it passed.

$ gradle test --tests '*issue87Test'

BUILD SUCCESSFUL in 1s

But wehn I execute the collection of tests under the base subproject, the issue87Test failed.

:~/github/materialstore/base (Issue350 *)
$ gradle test --tests '*.reduce.*Test'

> Task :base:test

MaterialProductGroup_issue87Test > test_getQueryOnMetadataList() FAILED
    org.opentest4j.AssertionFailedError at MaterialProductGroup_issue87Test.java:111

59 tests completed, 1 failed

This finding suggests that some other test under the com.kazurayam.materialstore.base.reduce package is disturbing the issue87Test.

Which one?

[kazurayam]

I changed the base/build.gradle as

test {
    filter {
        includeTestsMatching("*reduce.MaterialProductGroup_BuilderTest")
        includeTestsMatching("*reduce.MaterialProductGroup_issue87Test")
    }
}

then I tried

:~/github/materialstore/base (Issue350 *)
$ gradle clean test

> Task :base:test

MaterialProductGroup_issue87Test > test_getQueryOnMetadataList() FAILED
    org.opentest4j.AssertionFailedError at MaterialProductGroup_issue87Test.java:111

8 tests completed, 1 failed

This proved that MaterialProductGroup_BuilderTest is distubing the MaterialProductGroup_issue87Test.

[kazurayam]

Then, my next question is, how MaterialProductGroup_BuilderTest is distubing the MaterialProductGroup_issue87Test.

[kazurayam]

With the build.gradle file having

test {
    filter {
        includeTestsMatching("*reduce.MaterialProductGroup_BuilderTest")
        includeTestsMatching("*reduce.MaterialProductGroup_issue87Test")
    }
}

I did clean and test --- it failed.

$ gradle clean test

> Task :base:test

MaterialProductGroup_issue87Test > test_getQueryOnMetadataList() FAILED
    org.opentest4j.AssertionFailedError at MaterialProductGroup_issue87Test.java:111

8 tests completed, 1 failed

> Task :base:test FAILED

Soon after this, I tried test without clean --- i passed.

:~/github/materialstore/base (Issue350 *)
$ gradle test

BUILD SUCCESSFUL in 5s
10 actionable tasks: 2 executed, 8 up-to-date

Then I tried clean and test again --- it failed.

:~/github/materialstore/base (Issue350 *)
$ gradle clean test

> Task :base:test

MaterialProductGroup_issue87Test > test_getQueryOnMetadataList() FAILED
    org.opentest4j.AssertionFailedError at MaterialProductGroup_issue87Test.java:111

8 tests completed, 1 failed

> Task :base:test FAILED

I tried this sequence repeatedly. I could reproduce this result.

There must be a clue here.

[kazurayam]

I changed the issue87Test so that it draws the diagram of the MaterialProductGroup object:

    @Test
    public void test_getQueryOnMetadataList() throws MaterialstoreException {

        List<QueryOnMetadata> queryList = mpg.getQueryOnMetadataList();

        System.out.println("[test_getQueryOnMetadataList]");
        for (int i = 0; i < queryList.size(); i++) {
            System.out.println(String.format("queryList.get(%d): %s", i, queryList.get(i).toString()));
        }

        // https://github.com/kazurayam/materialstore/issues/351
        // generate a diagram of the MaterialProductGroup object
        // save the image into a new folder
        String dot = DotGenerator.generateDot(mpg);
        BufferedImage bi = DotGenerator.toImage(dot);
        store.write(jobName, JobTimestamp.now(), FileType.PNG, Metadata.builder().build(), bi);

        // Assert
        Assertions.assertEquals(8, queryList.size());

When I tried $ gradle test, it passed; it produced the following image:

When I tried $ gradle clean test, it failed; it produced the following image:

Well, difficult to understand what's going on...

[kazurayam]

I wrote that "$ gradle test" passes, but actually not necessarily so.

:~/github/materialstore/base (Issue350 *)
$ gradle test

> Task :base:test

MaterialProductGroup_issue87Test > test_getQueryOnMetadataList() FAILED
    org.opentest4j.AssertionFailedError at MaterialProductGroup_issue87Test.java:130

8 tests completed, 1 failed

I missed the clue.

[kazurayam]

This issue is diffuclt. I seem to be unable to fix this. I will disable 2 test cases tentatively, so that I can continue works for other topis.