jdereg / json-io

Convert Java to JSON. Convert JSON to Java. Pretty print JSON. Java JSON serializer. Deep copy Java object graphs.
Apache License 2.0
331 stars 116 forks source link

Test Fix: Fix four tests that can unintentionally fail due to Inconsistent JSON String Order #301

Closed Jake-WangZhi closed 1 month ago

Jake-WangZhi commented 1 month ago

I found a few more tests that can fail unintentionally for the same reason mentioned in this PR. The JSON string may not maintain a consistent order when converting an object to JSON, which can result in an AssertionError when comparing the converted JSON with the expected JSON.

Below are where they fails under NonDex on

Click on to see more details on the error message when running each test ``` [INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.cedarsoftware.io.AtomicBooleanTest [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.611 s <<< FAILURE! - in com.cedarsoftware.io.AtomicBooleanTest [ERROR] com.cedarsoftware.io.AtomicBooleanTest.testAssignAtomicBoolean Time elapsed: 0.588 s <<< FAILURE! org.opentest4j.AssertionFailedError: expected: "{"@type":"com.cedarsoftware.io.AtomicBooleanTest$TestAtomicBooleanField","value":true,"nullValue":null,"strValue":true,"emptyStrValue":false,"objValue":false,"values":[false,null,true,true]}" but was: "{"@type":"com.cedarsoftware.io.AtomicBooleanTest$TestAtomicBooleanField","value":true,"strValue":true,"emptyStrValue":false,"nullValue":null,"objValue":false,"values":[false,null,true,true]}" at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at com.cedarsoftware.io.AtomicBooleanTest.testAssignAtomicBoolean(AtomicBooleanTest.java:62) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] AtomicBooleanTest.testAssignAtomicBoolean:62 expected: "{"@type":"com.cedarsoftware.io.AtomicBooleanTest$TestAtomicBooleanField","value":true,"nullValue":null,"strValue":true,"emptyStrValue":false,"objValue":false,"values":[false,null,true,true]}" but was: "{"@type":"com.cedarsoftware.io.AtomicBooleanTest$TestAtomicBooleanField","value":true,"strValue":true,"emptyStrValue":false,"nullValue":null,"objValue":false,"values":[false,null,true,true]}" [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 -------------------------------------------------------------------------------------------------------------------------------------------- [INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.cedarsoftware.io.CollectionTests [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.491 s <<< FAILURE! - in com.cedarsoftware.io.CollectionTests [ERROR] com.cedarsoftware.io.CollectionTests.testEnumsInsideOfACollection_whenWritingAsObject_withPrivateMembersIncluded Time elapsed: 0.45 s <<< FAILURE! org.opentest4j.AssertionFailedError: expected: <{"@type":"ArrayList","@items":[{"@type":"com.cedarsoftware.io.CollectionTests$TestEnum4","age":21,"foo":"bar","name":"B"}]}> but was: <{"@type":"ArrayList","@items":[{"@type":"com.cedarsoftware.io.CollectionTests$TestEnum4","foo":"bar","age":21,"name":"B"}]}> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197) at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182) at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177) at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141) at com.cedarsoftware.io.CollectionTests.testEnumsInsideOfACollection_whenWritingAsObject_withPrivateMembersIncluded(CollectionTests.java:396) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] CollectionTests.testEnumsInsideOfACollection_whenWritingAsObject_withPrivateMembersIncluded:396 expected: <{"@type":"ArrayList","@items":[{"@type":"com.cedarsoftware.io.CollectionTests$TestEnum4","age":21,"foo":"bar","name":"B"}]}> but was: <{"@type":"ArrayList","@items":[{"@type":"com.cedarsoftware.io.CollectionTests$TestEnum4","foo":"bar","age":21,"name":"B"}]}> [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 [INFO] -------------------------------------------------------------------------------------------------------------------------------------------- [INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.cedarsoftware.io.NoTypeTest [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.668 s <<< FAILURE! - in com.cedarsoftware.io.NoTypeTest [ERROR] com.cedarsoftware.io.NoTypeTest.testNoType Time elapsed: 0.607 s <<< FAILURE! org.opentest4j.AssertionFailedError: expected: <{"name":"Zeus","things":[1,2,"3","4",-84243801600000,"Hello","com.cedarsoftware.io.NoTypeTest$Junk"],"namesToAge":{"Appollo":2500,"Hercules":2489,"Poseidon":"2502","Aphrodite":"2499.0","Zeus":-84243801600000},"stuff":[1,2,"3","4",-84243801600000,"Hello","com.cedarsoftware.io.NoTypeTest$Junk"]}> but was: <{"stuff":[1,2,"3","4",-84243801600000,"Hello","com.cedarsoftware.io.NoTypeTest$Junk"],"things":[1,2,"3","4",-84243801600000,"Hello","com.cedarsoftware.io.NoTypeTest$Junk"],"namesToAge":{"Appollo":2500,"Hercules":2489,"Poseidon":"2502","Aphrodite":"2499.0","Zeus":-84243801600000},"name":"Zeus"}> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197) at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182) at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177) at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141) at com.cedarsoftware.io.NoTypeTest.testNoType(NoTypeTest.java:267) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] NoTypeTest.testNoType:267 expected: <{"name":"Zeus","things":[1,2,"3","4",-84243801600000,"Hello","com.cedarsoftware.io.NoTypeTest$Junk"],"namesToAge":{"Appollo":2500,"Hercules":2489,"Poseidon":"2502","Aphrodite":"2499.0","Zeus":-84243801600000},"stuff":[1,2,"3","4",-84243801600000,"Hello","com.cedarsoftware.io.NoTypeTest$Junk"]}> but was: <{"stuff":[1,2,"3","4",-84243801600000,"Hello","com.cedarsoftware.io.NoTypeTest$Junk"],"things":[1,2,"3","4",-84243801600000,"Hello","com.cedarsoftware.io.NoTypeTest$Junk"],"namesToAge":{"Appollo":2500,"Hercules":2489,"Poseidon":"2502","Aphrodite":"2499.0","Zeus":-84243801600000},"name":"Zeus"}> [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 [INFO] -------------------------------------------------------------------------------------------------------------------------------------------- [INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.cedarsoftware.io.PrettyPrintTest [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.554 s <<< FAILURE! - in com.cedarsoftware.io.PrettyPrintTest [ERROR] com.cedarsoftware.io.PrettyPrintTest.testPrettyPrint Time elapsed: 0.517 s <<< FAILURE! org.opentest4j.AssertionFailedError: Expecting actual: "{ "@type":"com.cedarsoftware.io.PrettyPrintTest$Nice", "items":{ "@type":"ArrayList", "@items":[ "One", 1, { "@type":"Integer", "value":1 }, true ] }, "dictionary":{ "@type":"LinkedHashMap", "grade":"A", "price":100.0, "bigdec":{ "@type":"BigDecimal", "value":"3.141592653589793238462643383" } }, "name":"Louie" }" to be equal to: "{ "@type":"com.cedarsoftware.io.PrettyPrintTest$Nice", "name":"Louie", "items":{ "@type":"ArrayList", "@items":[ "One", 1, { "@type":"Integer", "value":1 }, true ] }, "dictionary":{ "@type":"LinkedHashMap", "grade":"A", "price":100.0, "bigdec":{ "@type":"BigDecimal", "value":"3.141592653589793238462643383" } } }" when ignoring newlines (\n, \r\n). at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at com.cedarsoftware.io.PrettyPrintTest.testPrettyPrint(PrettyPrintTest.java:58) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] PrettyPrintTest.testPrettyPrint:58 Expecting actual: "{ "@type":"com.cedarsoftware.io.PrettyPrintTest$Nice", "items":{ "@type":"ArrayList", "@items":[ "One", 1, { "@type":"Integer", "value":1 }, true ] }, "dictionary":{ "@type":"LinkedHashMap", "grade":"A", "price":100.0, "bigdec":{ "@type":"BigDecimal", "value":"3.141592653589793238462643383" } }, "name":"Louie" }" to be equal to: "{ "@type":"com.cedarsoftware.io.PrettyPrintTest$Nice", "name":"Louie", "items":{ "@type":"ArrayList", "@items":[ "One", 1, { "@type":"Integer", "value":1 }, true ] }, "dictionary":{ "@type":"LinkedHashMap", "grade":"A", "price":100.0, "bigdec":{ "@type":"BigDecimal", "value":"3.141592653589793238462643383" } } }" when ignoring newlines (\n, \r\n). [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 [INFO] ```

To reproduce each, run these at the root directory:

mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=com.cedarsoftware.io.AtomicBooleanTest#testAssignAtomicBoolean -DnondexRuns=10
mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=com.cedarsoftware.io.CollectionTests#testEnumsInsideOfACollection_whenWritingAsObject_withPrivateMembersIncluded -DnondexRuns=10
mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=com.cedarsoftware.io.NoTypeTest#testNoType -DnondexRuns=10
mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=com.cedarsoftware.io.PrettyPrintTest#testPrettyPrint -DnondexRuns=10

(Note: The failing test might not be seen if every test happens to have the JSON string in the correct order. Try running it several times or increase the the number of runs with -DnondexRuns= to reproduce the issue.)

The log output for each test can be found here for your reference: mvn-nondex-testAssignAtomicBoolean-1729687816.log mvn-nondex-testEnumsInsideOfACollection_whenWritingAsObject_withPrivateMembersIncluded-1729688247.log mvn-nondex-testNoType-1729686012.log mvn-nondex-testPrettyPrint-1729687592.log

To resolve this, We can apply the same solution used in the previous PR by converting the JSON string to a JSON element using JsonParser. This way, the order of the elements won’t matter, and we can focus on testing whether the content of the JSON matches the expected output.

After applying the fix, the test should now pass with NonDex as expected:

[INFO] *********
[INFO] All tests pass without NonDex shuffling
[INFO] ####################
[INFO] Across all seeds:
[INFO] Test results can be found at: 
[INFO] file:///home/jakew4/json-io/.nondex/Edw3hT+yzFguWpy1wrFFrTkO8wygCDeeR4dxgTkq2qQ=/test_results.html
[INFO] [NonDex] The id of this run is: Edw3hT+yzFguWpy1wrFFrTkO8wygCDeeR4dxgTkq2qQ=
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  23.127 s
[INFO] Finished at: 2024-10-23T10:09:59-05:00
[INFO] ------------------------------------------------------------------------

Please let me know if this approach works for you. If not, I'm happy to discuss alternatives and am willing to spend more time to address the test in the way you'd prefer. Thank you!