search

extent-framework / extentreports-cucumber4-adapter

Cucumber4 Adapter for Extent Framework
http://extentreports.com/docs/versions/4/java/cucumber2.html
Apache License 2.0
39 stars 22 forks source link

CucumberOptions(Strict=true) not failing the Undefined Steps present in Feature File #55

Open suri1990 opened 4 years ago

suri1990 commented 4 years ago

Steps To Reproduce :

-- Created a sample feature file in Java maven project. -- For few steps in feature for a particular test scenario , didn't defined the step definition code. -- Using @CucumberOptions(strict=true) in Runner class with other mandatory options as well. -- Run the maven project from Eclipse IDE.

Actual Result : Undefined Steps are not failing and instead showing pass in HTML report.(Although getting warning in eclipse console to implement undefined steps).

Expected Result : As per strict= true option, undefined steps should fail.

Note : Even with strict=false, undefined tests are getting PASSED instead of getting skipped

Below are some required details -

OS : Windows 8.1 Java Version : 8 Cucumber Version : 5.6.0 Maven version : 3.3.9 Extent Report Adapter : ExtentReport Cucumber5 Adapter

eclipse_console.txt extent_report feature_file step_definiton_file.txt testrunner_class

anshooarora commented 4 years ago

@suri1990 This issue has been fixed in cucumber4 adapter v1.0.12. Please update your version and try again.

suri1990 commented 4 years ago

@anshooarora , I am using all latest Jars of cucumber(version 5.x.x) in my project and correspondingly cucumber5 adapter. Is this an issue with v5.x.x of cucumber jars and cucumber5 adapter for the wrong display of extent report? If Yes, then do I need to change all Jars of cucumber to v4.x.x and use cucumber4 adapter?

grasshopper7 commented 4 years ago

@suri1990 I will change this in the version 5 adapter and upload. The status for missing step definitions is now "undefined".

grasshopper7 commented 4 years ago

@anshooarora Gave version 1.0.12, with cucumber 4, a try with scenarios having no step definitions. It is showing these scenarios and steps as passed in the Spark report.

Used the feature file from the test resources folder in this repo - shopping.feature

Below are more details as observed from the Spark report.

  1. All step definitions are absent - In the dashboard, the feature, scenario and steps are shown as passed. dashboard In the tests tab, the test and steps are shown as passed. tests In the tags tab, test is also shown as passed. tabs If the scenarios have hooks (without scenario.write()) the same results are displayed.

2.` If the scenarios have hooks with scenario.write() the dashboard shows the same as in point 1 except for the steps pie. The hooks are included in the steps pie. dashboard-steps The hooks are included in the tests tag. tests The tags tab display is same as point 1.

  1. When step definitions are added for the steps except the first step, the features and scenario pie chart show skipped, The steps pie chart show 1 passed but all steps should be skipped. dashboard In the tests tab the first step is sown as passed while all steps should be marked skipped. tests

For more details refer to this excel - SkipCombos.xlsx

The skipped issue can be fixed by adding the "undefined" case, similar to the 'skipped' and 'pending' cases, in the updateResult(Result result) method of the ExtentCucumberAdapter class.

Are the display of hooks with scenario.write in the steps pie and test tab are working as per design?

On the point of strict=true, the adapter does not differentiate between the two options, it simply does not take it into consideration. My understanding is that the true setting should fail the overall run or build, rather than a specific scenarios and features. Can you clarify on this?

Thanks

anshooarora commented 4 years ago

@grasshopper7 thank you for your research. The adapter needs to implement StrictAware to consume the status of strict flag. Please review the commit https://github.com/extent-framework/extentreports-cucumber4-adapter/commit/7b027b95d260fe5ced254b55ce67f7e7df746235.

@suri1990 Published snapshot release to nexus, please use 1.1.0-SNAPSHOT. You will have to ensure SNAPSHOTS are enabled from settings.xml.

suri1990 commented 4 years ago

@anshu - Extent report looks fine to me with adapter5 v1.1.0. Just a query whether @After/Before/BeforeStep/AfterStep are counted in steps in extent report?

@grasshopper7 - please check at your end as well.

Thanks Surender

On Tue, 21 Apr 2020, 5:49 am Anshoo Arora, notifications@github.com wrote:

Reopened #55 https://github.com/extent-framework/extentreports-cucumber4-adapter/issues/55 .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/extent-framework/extentreports-cucumber4-adapter/issues/55#event-3254037836, or unsubscribe https://github.com/notifications/unsubscribe-auth/APGQD2ON3LC5LTDNNNIEQIDRNTQ7TANCNFSM4MJPAKLA .

grasshopper7 commented 4 years ago

@suri1990 I will be updating the adpater v5 with minor changes to match the latest code in version 4 adapter and upload to nexus, possibly tonight. Will update here when done.

suri1990 commented 4 years ago

Sure. Just had one query mentioned above whether @ After/Before/BeforeStep/AfterStep are counted in steps in extent report?

Thanks Surender

On Tue, 21 Apr 2020, 5:55 pm grasshopper, notifications@github.com wrote:

@suri1990 https://github.com/suri1990 I will be updating the adpater v5 with minor changes to match the latest code in version 4 adapter and upload to nexus, possibly tonight. Will update here when done.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/extent-framework/extentreports-cucumber4-adapter/issues/55#issuecomment-617146970, or unsubscribe https://github.com/notifications/unsubscribe-auth/APGQD2L7IVTMPE3FABEEBMLRNWGCLANCNFSM4MJPAKLA .

grasshopper7 commented 4 years ago

@anshooarora Tested it out with some of the test cases from yesterday and it looks good. The skip steps are displayed correctly and the strict="true" option fails the skipped steps.

One query from earlier - Hooks with scenario.write are included in the steps count. What is the logic behind this?

Second query related to tags tab - Have 2 feature files. One named 'shopping' containing 2 scenarios. Another 'Dummy' with 1 scenario. The count of tests on the left is '5', shouldn't this be '3'. tagstab

Thanks.

anshooarora commented 4 years ago

@grasshopper7 this is nothing to do with the adapter but the Extent API itself. It is configured to account for each "event" that is raised - in our case these are scenario.handleWrite which are children of hook steps (@AfterStep @BeforeStep etc). To account for this behavior, a change would have to be made from the API, while being generic as the API is not tied to any one type of runner/framework. Marking this as research.