Open itkhanz opened 4 months ago
I have tried to resolve this issue by providing a path to merged merged junit xml report generated by org.testng.reporters.JUnitXMLReporter as:
- task: Gradle@3
displayName: 'Execute gradle test'
continueOnError: true
inputs:
gradleWrapperFile: 'gradlew'
options: ${{ variables.gradleParameters }}
tasks: test --info
testResultsFiles: "**/Gradle suite/Gradle test.xml" #path to merged JUNIT XML report generated by TESTNG
It resolves the issue of total run duration but causes another problem i.e. it does not ignore the tests that were not run and are marked as ignored in the XML. For example as you can see in the below merged xml output:
This test was not run during this build but test summary tab also includes this test as well as all others that were filtered out in the list of tests that were passed and assign zero duration to them which causes incorrect number of total tests.
Output should be like this in Test summary tab in Azure DevOps pipeline:
> Task :test FAILED
4 passing (8m 36s)
3 failing
---------------------------------------------------------------
| Results: FAILURE (7 tests, 4 passed, 3 failed, 0 skipped) |
---------------------------------------------------------------
Test report disabled, omitting generation of the HTML test report.
7 tests completed, 3 failed
but it shows all 68 tests (including those that were not run as tests that passed). Here is the debug output of Azure task:
##[debug]Object of JUnitResultParser created.
##[debug]Entering ParseTestResultFiles
##[debug]runContext.ReleaseURI is null
##[debug]runContext.ReleaseEnvironmentUri is null
##[debug]Reading test results from file '/agent/_work/2/s/build/testng/Gradle suite/Gradle test.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 68.
##[debug]Leaving ParseTestResultFiles
The merged junit xml report looks like as follows:
<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated by org.testng.reporters.JUnitXMLReporter -->
<testsuite ignored="61" hostname="7f87e08ea7dc" failures="3" tests="7" name="Gradle test" time="513.435" errors="0" timestamp="2024-05-12T15:17:09 UTC">
<testcase classname="com.dcs.appium.dcsapps.Tests.Favourites.FavouritesTest" name="VSDS_2406_VSDS_2408_Check_Adding_POI_As_Favourite_Then_Delete_It" time="3.361">
.
.
.
.
<testcase classname="com.dcs.appium.dcsapps.Tests.PoiDetailView.Reviews.Comment.VSDS_7324_Check_Adding_Review_With_Comments_via_Review_Button" name="VSDS_7324_Check_Adding_Review_With_Comments_via_Review_Button" time="0.0">
<ignored/>
</testcase> <!-- VSDS_7324_Check_Adding_Review_With_Comments_via_Review_Button -->
I see this could be the reason that Azure fails to recognize that attributes in testsuite tag which states total tests 7, failed 3, and ignored 61.
Hi @itkhanz thanks for reporting! We are working on more prioritized issues at the moment, but will get back to this one soon.
New issue checklist
Task name
Gradle@3 - Gradle v3 task
Task version
3
Issue Description
I am running my TestNG test suite in Azure pipelines via gradle task as
gradle test
. Since this task has by defaultpublishJUnitResults
property to true so it automatically publishes the test results after build. I am specifying the location of Junit XML reports generated by TestNG for this task to scan and then process test results. This is being done intentionally as be default gradle also produces Junit XML reports as**/TEST-*.xml
which I have disabled to avoid test count duplication and pointed this task to load Junit XML reports generated by TestNG instead.Although it correctly reads the test count and individual test durations but it shows incorrect total run duration as you can see in the screenshot below:
The correct test duratation is 8.26 minutes (sum of individual tests) as it can also be seen in the gradle logs, pipeline summary tab, as well as in the testng-results.xml file:
Job Summary
Gradle logs
testng-results.xml
Total test suite duration is 495755 ms = 8.26 minutes
Actual Result: Total run duration is calculated only from the latest XML result file parsed. In this case, whatever is the last test suite (XML file) detected, it will become total test run duration. As the run duration of last read result is 2m 4s, hence the total run duration is shown as 2m 4s.
Expected Result: Total run duration should be calculated as the sum of time durations of detected testsuites.
TestNG 7.10.1 Gradle 8.7
Environment type (Please select at least one enviroment where you face this issue)
Azure DevOps Server type
Azure DevOps Server (Please specify exact version in the textbox below)
Azure DevOps Server Version (if applicable)
No response
Operation system
MacOS Sonoma 14.4.1
Relevant log output
Full task logs with system.debug enabled
Repro steps