Found some Non deterministic Tests

Describe the bug

Possible test failures in module manifold-deps-parent/manifold-json-test and manifold-deps-parent/manifold-templates-test

1. Tests that could fail

Class Name1: manifold.api.json.RequesterTest Test Names: httpGetRequestWithParameterizedUrlSuffixWithParams, httpGetRequestWithParams, httpPostRequestWithParameterizedUrlSuffixWithParams, httpPostRequestWithParams

Class Name2: manifold.templates.directives.ParamsTest Test Name: importedParamWorks

Class Name3: manifold.templates.directives.SectionTest Test Name: includeNestedSectionTest

2. Why the tests might fail?

The tests httpGetRequestWithParameterizedUrlSuffixWithParams, httpGetRequestWithParams, httpPostRequestWithParameterizedUrlSuffixWithParams, httpPostRequestWithParams in RequesterTest.java are creating objects for the Requester class. Requester.java returns a HashMap https://github.com/manifold-systems/manifold/blob/4f8d5c73cf0e2999d811f80a2101dc3723b273f5/manifold-deps-parent/manifold-json-rt/src/main/java/manifold/json/rt/api/Requester.java#L164 According to the official documentation, HashMap in Java does not maintain the order of the elements inserted. Because we insert two parameters here,

.withParam( "foo", "bar" )
.withParam( "abc", "8" );

This order is not guaranteed while checking. Since there is no guarantee in the order of exception types returned, the order might change, resulting in the test failure.

ParamsTest.java uses HashSet and then makes an assert check here

https://github.com/manifold-systems/manifold/blob/4d5d176027a7235cf5c11cf2aa2191a426ae0637/manifold-deps-parent/manifold-templates-test/src/test/java/manifold/templates/directives/ParamsTest.java#L26

According to the official documentation, HashSet does not guarantee the order of the elements in which they are inserted. Because of this, the assert check fails sometimes, making the test non-deterministic.

SectionTest.java has an assert check here

https://github.com/manifold-systems/manifold/blob/4d5d176027a7235cf5c11cf2aa2191a426ae0637/manifold-deps-parent/manifold-templates-test/src/test/java/manifold/templates/directives/SectionTest.java#L62

The render method here, changes the order of the elements, making the test non-deterministic.

Solution

In RequesterTest.java, the assert checks should pass as we're inserting two parameters and checking them. However, because the HashMap does not maintain order, the test becomes non-deterministic. Instead of doing assertEquals, we can use assertTrue and check for both the possible orders. This way, we check if two parameters exist (in any order). (foo=bar, abc=8 and abc=8, foo=bar). According to the official documentation, LinkedHashMap preserves the order in which the parameters are inserted. In ParamsTest.java, by converting the HashSet to LinkedHashSet, we fix the randomness of the importedParamWorks test. Because LinkedHashSet preserves the order of the elements in which they are inserted. In SectionsTest.java, because there are only 3 lines, we can do assertTrue for all three lines and check if all three exist. This way we don't have to be worried about the order of the lines.

To Reproduce

Steps to reproduce the behavior:

I used an open-source tool called NonDex to detect the assumption by shuffling the order of returned exception types. Running the following commands will test the aforementioned operation

Clone the Repo

https://github.com/manifold-systems/manifold

Compile the module

mvn install -pl manifold-deps-parent/manifold-json-test -am -DskipTests

(Optional) Run the unit test

mvn -pl manifold-deps-parent/manifold-json-test test -Dtest=manifold.api.json.RequesterTest#httpGetRequestWithParameterizedUrlSuffixWithParams

Run the unit test using NonDex

mvn -pl manifold-deps-parent/manifold-json-test edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=manifold.api.json.RequesterTest#httpGetRequestWithParameterizedUrlSuffixWithParams

Desktop (please complete the following information):

OS Type & Version: macOS 13.1 (22C65)
Java/JDK version: 8
IDE version (IntelliJ IDEA or Android Studio): IntelliJ IDEA

Stack trace

[INFO] Running manifold.api.json.RequesterTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.098 s <<< FAILURE! - in manifold.api.json.RequesterTest
[ERROR] httpPostRequestWithParams(manifold.api.json.RequesterTest)  Time elapsed: 0.096 s  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[foo=bar&abc=8]> but was:<[abc=8&foo=bar]>
        at manifold.api.json.RequesterTest.httpPostRequestWithParams(RequesterTest.java:50)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   RequesterTest.httpPostRequestWithParams:50 expected:<[foo=bar&abc=8]> but was:<[abc=8&foo=bar]>

[ERROR] importedParamWorks(manifold.templates.directives.ParamsTest)  Time elapsed: 0.044 s  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[123]> but was:<[312]>
        at manifold.templates.directives.ParamsTest.importedParamWorks(ParamsTest.java:30)

[ERROR] includeNestedSectionTest(manifold.templates.directives.SectionTest)  Time elapsed: 0.05 s  <<< FAILURE!
org.junit.ComparisonFailure: 
expected:<...2 style="font-size: [1">Font size: 1</h2>
        <h2 style="font-size: 2">Font size: 2]</h2>
        <h2 st...> but was:<...2 style="font-size: [2">Font size: 2</h2>
        <h2 style="font-size: 1">Font size: 1]</h2>
        <h2 st...>
        at manifold.templates.directives.SectionTest.includeNestedSectionTest(SectionTest.java:62)

manifold-systems / manifold