Flaky Test found using NonDex by running the command -
mvn -pl cloudslang-runtime edu.illinois:nondex-maven-plugin:1.1.2:nondex -Dtest=io.cloudslang.lang.runtime.bindings.scripts.ScriptExecutorTest#testExternalPythonValid
The logged failure for the io.cloudslang.lang.runtime.bindings.scripts.ScriptExecutorTest.testExternalPythonValid -
[ERROR] Failures:
[ERROR] ScriptExecutorTest.testExternalPythonValid:128 arrays first differed at element [0]; expected:<input[1]> but was:<input[2]>
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
Investigation
The test fails at ScriptExecutorTest.java:128 with a comparison error while comparing an expected keyset array and the result from the captured keyset array. The keySet function of the Map class makes no guarantees as to the iteration order of the keys in the Map object. This makes the test outcome non-deterministic, and the test fails whenever keySet returns a mismatch in order of the keys both the Maps. To fix this, the expected and actual maps' keys should be checked in a more deterministic way so that the assertions do not fail.
Fix
Expected and actual arrays can be compared without making assumptions about the order of the keys in these Maps. As the Matchers.containsInAnyOrder function compares the values inside the set of keys without needing order, the test becomes deterministic and ensures that the flakiness from the test is removed.
Description
Flaky Test found using NonDex by running the command -
mvn -pl cloudslang-runtime edu.illinois:nondex-maven-plugin:1.1.2:nondex -Dtest=io.cloudslang.lang.runtime.bindings.scripts.ScriptExecutorTest#testExternalPythonValid
The logged failure for the io.cloudslang.lang.runtime.bindings.scripts.ScriptExecutorTest.testExternalPythonValid -
Investigation
The test fails at ScriptExecutorTest.java:128 with a comparison error while comparing an expected keyset array and the result from the captured keyset array. The keySet function of the Map class makes no guarantees as to the iteration order of the keys in the Map object. This makes the test outcome non-deterministic, and the test fails whenever keySet returns a mismatch in order of the keys both the Maps. To fix this, the expected and actual maps' keys should be checked in a more deterministic way so that the assertions do not fail.
Fix
Expected and actual arrays can be compared without making assumptions about the order of the keys in these Maps. As the Matchers.containsInAnyOrder function compares the values inside the set of keys without needing order, the test becomes deterministic and ensures that the flakiness from the test is removed.
The PR does not introduce a breaking change.
Bharati Kulkarni bmk15897@gmail.com