Reporting - listing flaky tests

[lukasz-jarocki-sonarsource]

Hi,

I wonder if as of today there is any easy way to list all the flaky tests in the file (at the specified path)?

Our usecase is that we have large codebase and we run unit tests in thousands of packages. Each run for each class creates default XML file with the report, however:

• we need to parse each of these XML files to find flaky tests
• there is no clear indication in the XML files which tests are flaky - we need to parse XML and look for a test that has a failure message and at the same we need to find entry for this test that doesn't have failure message.

I wonder if the plugin could automatically list all the flaky tests across all the packages on the specified path.

[pshevche]

Hi @lukasz-jarocki-sonarsource, Thank you for reaching out. Have you had a chance to explore Gradle Build Scans? They provide reacher test views, which, among other things, allow you to filter tests by outcome. So, for your use case, you would see all flaky tests in a module/package/class of choice. We don't plan to improve the reporting of this plugin taking into account that there is a more superior alternative out there.

Here is an example: https://scans.gradle.com/s/pyl5kd4heuwpm/tests/overview See how I have "many" test classes with some tests being flaky:

I can also filter only for flaky tests, which were successfully retried by this plugin:

The public scans.gradle.com instance does not have a full set of visualization tools for flaky tests. There is an option to set up a dedicated instance for your project. Here you could see what else we have to offer in the area of flaky tests management.

Let me know if you have any follow-up questions.

[lukasz-jarocki-sonarsource]

Hi, yes, we have evaluated build scans and unfortunately due to internal policy we can't use it (as it transmits information to remote server). And yes, we are aware of Gradle Enterprise.