Failed scenarios not correct in tags

extent-framework / extentreports-cucumber4-adapter

Cucumber4 Adapter for Extent Framework

http://extentreports.com/docs/versions/4/java/cucumber2.html

Apache License 2.0

39 stars 22 forks source link

Failed scenarios not correct in tags #50

Open zuzeac opened 4 years ago

zuzeac commented 4 years ago

Hi, I just observed some issues on the library,

the failed number of scenarios is wrong in the tags section. For e.g.: I had 1 scenario failed for tag @world-shop-catalog-feature, but it appears to be 2 fails in tags section from Dashboard.
if for some reason appium server is failing, or a device is disconnected , the next scenarios will be skipped, but they won't record in the report. Maybe because the skipped case is ignored?

version - 1.0.11 Spark report

anshooarora commented 4 years ago

@zuzeac

I am able to reproduce this
The 2 cases are grouped, so both "skipped" and "pending" will go inside the same block of code.

anshooarora commented 4 years ago

Partial fix was on the API side https://github.com/extent-framework/extentreports-java/issues/139

zuzeac commented 4 years ago

Hi @anshooarora , thank you for your prompt replay and fix. for case 2, both skipped and pending will go inside the same block of code, but they won't be displayed in the report. And that's the issue here. How can we add them in the report? When I was using an old version of extent report, not cucumber, then all skipped scenarios were recorded in the report.

anshooarora commented 4 years ago

They should also be logged. Can you check with <version>1.0.12-SNAPSHOT</version>?

zuzeac commented 4 years ago

I just checked, by using 1.0.12-SNAPSHOT, but i do not see any fix, the Skipped scenario is not logged.

anshooarora commented 4 years ago

@zuzeac Please create a small project reproducing the issue so I can see the behavior.

Did you check the below as per your original post:

the failed number of scenarios is wrong in the tags section. For e.g.: I had 1 scenario failed for tag @world-shop-catalog-feature, but it appears to be 2 fails in tags section from Dashboard.

Is this working?

EDIT: Please make sure to use the latest snapshot and force refresh the project. If you are using Eclipse, you need to check the box below:

PS. This is what I see in the latest:

zuzeac commented 4 years ago

the fix first issue seems to not work, 1.0.12-SNAPSHOT, do i need to update any other library?
second issue: Oh, i got it, you got it wrong, the skipped steps from a scenario are logged in the report. I was reffering to the following: let's assume I have a suite of 10 scenarios/tests, then at the 5th scenario something broken, test execution is interrupted (the device is disconnected) then the rest of the 5th scenarios are not logged. They should be logged as Skipped scenarios.

I'm using IntelliJ, As far as I can see the 1.0.12-SNAPSHOT was downloaded in my .m2 repo. I will try to create a small project to reproduce it

anshooarora commented 4 years ago

Everything should be logged, I am not sure why it isn't. For me, if 1 scenario fails, the rest still execute as they normally would, they are not skipped by default.

zuzeac commented 4 years ago

Hi, it might be my configurations, i have a combinations of testng with cucumber annotations. I create a small project and all work as expected. I will look further into this, to figure it out. You can close the ticket, if i'll find something i'll let you know. Thanks

anshooarora commented 4 years ago

Thanks, having a project to understand the scenario would help.

zuzeac commented 4 years ago

You can find a sample here https://github.com/zuzeac/appium-cucumber-extent-report.git . you'll need Appium installed and a phone connected to you machine. Just start the tests with mvn test, while the test starts, disconnect the device. And you will see that the next scenarios are not recorded. I think it's because the testng annotations *@beforeSuite, @BeforeMethod. When I used the cucumber annotations from hooks it worked ok.

anshooarora commented 4 years ago

@zuzeac I don't use Appium or other web test tools. I can test your code as-is without the need of any utility if you can help reproduce it. Also, there was a fix that went into the latest snapshot 1.1.0-SNAPSHOT which records pending tests as well. See here: https://github.com/extent-framework/extentreports-cucumber4-adapter/commit/7b027b95d260fe5ced254b55ce67f7e7df746235. Can you check it out?

zuzeac commented 4 years ago

hi @anshooarora, i test the 1.1.0-SNAPSHOT, I think it has something to do with the fact that i'm using TestNG annotations. When using cucumber annotation, the scenarios which did not run, will be recorded as failed When using TestNG annotations, the scenarios which did not run will not be recorded.

Cucumber:

TestNg: