Problem with iDFlakies Reporting Flaky Tests

[alperen21]

Hello,

I have been trying to utilize iDFlakies for detecting flaky tests but it doesn't seem to report flaky tests that are confirmed to be flaky and wondered if I am utilizing iDFlakies in an incorrect way.

I have created a dummy maven project to recreate the problem I have been facing. In this project there is a dummy test that is intentionally written to show flaky behavior:

`
@Test public void flakyTest() { double num = Math.random();

    if (num < 0.5) {
        assertTrue(false);
    }
    else {
        assertTrue(true);
    }
}

`

Multiple executions of mvn surefire:test confirms the flakiness. However when I run iDFlakies with mvn testrunner:testplugin -Ddetector.detector_type=random-class-method -Ddt.randomize.rounds=10 -Ddt.detector.original_order.all_must_pass=false

iDFlakies doesn't seem to report any flaky test. The log of execution is:

` (base) alperenyildiz@Alperen-MacBook-Pro dummyProject % mvn testrunner:testplugin -Ddetector.detector_type=random-class-method -Ddt.randomize.rounds=10 -Ddt.detector.original_order.all_must_pass=false [INFO] Scanning for projects... [INFO] [INFO] ----------------------< org.example:dummyProject >---------------------- [INFO] Building dummyProject 1.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] >>> testrunner-maven-plugin:1.2:testplugin (default-cli) > test-compile @ dummyProject >>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ dummyProject --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ dummyProject --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ dummyProject --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /Users/alperenyildiz/Desktop/dummyProject/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ dummyProject --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] <<< testrunner-maven-plugin:1.2:testplugin (default-cli) < test-compile @ dummyProject <<< [INFO] [INFO] [INFO] --- testrunner-maven-plugin:1.2:testplugin (default-cli) @ dummyProject --- [INFO] Accessing cachePath: [INFO] No timeout specified. Using randomize.rounds: 10 [INFO] ROUNDS_CALCULATED: Giving org.example:dummyProject:1.0-SNAPSHOT 10 rounds to run for random-class-method [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Getting original results (1 tests). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Created dependent test detector (class edu.illinois.cs.dt.tools.detection.detectors.RandomDetector). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: Verified dummyTest.flakyTest, status: expected PASS, got ERROR [INFO] Found 0 tests in round 1 of 10 (0,0 seconds elapsed (1,7 total), 0,0 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 2 of 10 (1,7 seconds elapsed (2,8 total), 6,7 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 3 of 10 (2,8 seconds elapsed (4,1 total), 6,5 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 4 of 10 (4,1 seconds elapsed (4,9 total), 6,2 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 5 of 10 (4,9 seconds elapsed (5,7 total), 4,9 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 6 of 10 (5,7 seconds elapsed (6,5 total), 3,8 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 7 of 10 (6,5 seconds elapsed (7,3 total), 2,8 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 8 of 10 (7,3 seconds elapsed (8,1 total), 1,8 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 9 of 10 (8,1 seconds elapsed (8,9 total), 0,9 seconds remaining). [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Accessing cachePath: [INFO] Found 0 tests in round 10 of 10 (8,9 seconds elapsed (9,7 total), 0,0 seconds remaining).

[INFO] Found 0 tests, writing list to /Users/alperenyildiz/Desktop/dummyProject/.dtfixingtools/detection-results/list.txt and dt lists to /path/2/project/dummyProject/.dtfixingtools/detection-results/flaky-lists.json TRY_COPY_ALL_FAILING_TEST_OUTPUT

`

Zip file of the dummy project is included, any help would be highly appreciated.

Thank you in advance, dummyProject.zip

[august782]

Hi,

Thanks for trying iDFlakies. In this case, since you are running the random-class-method detector that changes test orders, it actually only tries to report the confirmed order-dependent flaky tests. This detector has a filter that reruns failed tests in the found passing and failing orders to ensure they are consistently passing and failing in those orders before keeping them. You can actually see under the generated .dtfixingtools/detection-results/random-class-method-confirmation-sampling/ directory JSON files corresponding to runs where the test fails (but it was expecting it to pass in the original order).

If you run using another detector, such as original (where it runs the original order many times), it would report such tests as flaky, as it does not try to confirm that they are order-dependent.

That said, I think you raise a good point that we are not properly reporting such non-order-dependent flaky tests when using the various reordering detectors. We should change the reported results to include these flaky tests as well as some marker on whether they are order-dependent or non-order-dependent.

[alperen21]

Hello,

Setting the detector type to original fixed the problem.

Thank you very much for the prompt reply,