search

UT-SE-Research / iDFlakies

29 stars 33 forks source link

Substantial performance difference between iDFlakies versions #31

Closed winglam closed 3 years ago

winglam commented 3 years ago

Between iDFlakies version 1.0.2 and 1.1.0-SNAPSHOT, some changes apparently cause iDFlakies to run much slower.

E.g., for alibaba/fastjson and iDFlakies version 1.0.2, running 55 tests take only 33s, while iDFlakies version 1.1.0-SNAPSHOT takes 4m47s. Both runs do not find any flaky tests.

Aliaba fastjson version: 65f65e1443484b91d427253e79ed305ca79d6d57 iDFlakies 1.1.0-SNAPSHOT: 2ea44c2040c152dba95cea31c9e79175c76053c2

original-order.log idflakies-1.1.0-SNAPSHOT.log idflakies-1.0.2.log

winglam commented 3 years ago

The cause for the performance difference is because the support for JUnit 5 required a check to see if a project would contain a mix of JUnit 4 and JUnit 5 tests, or just one or the other. https://github.com/idflakies/iDFlakies/blob/8c446cb7ba7a9708c10e5e7975bcda646e55a452/src/main/java/edu/illinois/cs/dt/tools/detection/DetectorPlugin.java#L219

No fix is deemed necessary as the check is needed to throw an exception if there are mixed tests.

Changes were introduced to prevent the locating of tests from running multiple times unnecessarily. https://github.com/idflakies/iDFlakies/commit/8c446cb7ba7a9708c10e5e7975bcda646e55a452