Flank / flank

:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab
https://firebase.community/
Apache License 2.0
672 stars 113 forks source link

Generate flaky results in HTML report #2368

Open AaronMT opened 1 year ago

AaronMT commented 1 year ago

As part of an (optional?) flag for Android, I would love to have an HTMLErrorReport report asset generated akin to the HTMLErrporReport HTML generated for failures (!matrices.isAllSuccessful())[1], will generate the same style report but with JUnit Test-cases that are identified as flaky (you can have a matrice with failures and flaky results, or all successful runs with one flaky, eg). Currently, I have to use JUnitParser (third party) Python library to parse the FullJUnitReport.xml file.

This would be a tremendous addition.

The <testcase> elements have an attribute flaky=true with the <testsuite> having the attribute flakes set non-zero.

I would simply want the name/classname attribute posted and the failure block and weblink in the generated report.

[1] https://github.com/Flank/flank/blob/8156e284776c8950707caa408910de7b0678f79d/test_runner/src/main/kotlin/ftl/reports/util/ReportManager.kt#L58