search

qmetry / qaf

Quality Automation Framework for web, mobileweb, mobile native and rest web-service using Selenium, webdrier, TestNG and Java Jersey
https://qmetry.github.io/qaf
MIT License
252 stars 137 forks source link

issue with dashboard report while execute failed-testcases more then one time #205

Open narendrarajput opened 6 years ago

narendrarajput commented 6 years ago

QAF Version

qaf and support v2.1.14-RC1

Scenario to reproduce

Step 1: While I'm executing the test lets say there is only one BDD file and only one scenario which is going to fail So first time if i see the execution in dashboard it is showing everything fine there. (count and scenario)

Step 2: If navigate to /target>surefire-reports and execute the testng-failed-qas.xml then after complete execution (test still failed) if view the report then it show the correct count and correct name of scenarios which failed this time as well.

Step 3: If again execute testng-failed-qas.xml and view the dashboard then it shows the correct count on overview but in detailed report it added the same failed testcases again(duplicate)

Note : If again run the same testng-failed-qas.xml then it show the same wrong details for scenario. Scenario observed with only one testcase if there is multiple testcases then its very hard to understand what's in the report

Expected behavior

It should show correct count and test scenario detail for each failed test execution summary

Actual behavior

Adding the same test scenario again if execute fail test from testng-failed-qas.xml more then once

Refer the snap

image

cjayswal commented 6 years ago

It looks same directory used for saving test results. This happens when you are using maven and instead of running maven test, running as TestNG through IDE.

narendrarajput commented 6 years ago

Yes I'm running failed test as TestNGSuite.

This is how i setup the things if i want to run failed test, than i have to do below changes in pom.xml

i have to map testng-failed-qas.xml in pom.xml e.g. <testSuiteFile>target/surefire-reports/testng-failed-qas.xml</testSuiteFile> and execute.

Is there another way to execute that testng-failed-qas.xml as Maven Test ?

cjayswal commented 6 years ago

You should refer maven documentation for that. Try by passing system property while executing maven.