CloudSlang / cloud-slang

CloudSlang Language, CLI and Builder
http://cloudslang.io
Apache License 2.0
236 stars 82 forks source link

Fixed Flaky Test caused by the Map.keySet function #1382

Open bmk15897 opened 2 years ago

bmk15897 commented 2 years ago

Description

Flaky Test found using NonDex by running the command - mvn -pl cloudslang-runtime edu.illinois:nondex-maven-plugin:1.1.2:nondex -Dtest=io.cloudslang.lang.runtime.bindings.scripts.ScriptExecutorTest#testExternalPythonValid

The logged failure for the io.cloudslang.lang.runtime.bindings.scripts.ScriptExecutorTest.testExternalPythonValid -

[ERROR] Failures: 
[ERROR]   ScriptExecutorTest.testExternalPythonValid:128 arrays first differed at element [0]; expected:<input[1]> but was:<input[2]>
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

Investigation

The test fails at ScriptExecutorTest.java:128 with a comparison error while comparing an expected keyset array and the result from the captured keyset array. The keySet function of the Map class makes no guarantees as to the iteration order of the keys in the Map object. This makes the test outcome non-deterministic, and the test fails whenever keySet returns a mismatch in order of the keys both the Maps. To fix this, the expected and actual maps' keys should be checked in a more deterministic way so that the assertions do not fail.

Fix

Expected and actual arrays can be compared without making assumptions about the order of the keys in these Maps. As the Matchers.containsInAnyOrder function compares the values inside the set of keys without needing order, the test becomes deterministic and ensures that the flakiness from the test is removed.

The PR does not introduce a breaking change.

Bharati Kulkarni bmk15897@gmail.com

bmk15897 commented 2 years ago

Can you please take a look at this pull request and approve it?

QueueUpX commented 2 years ago

@bmk15897 This will be merged when a new release is planned.