search

extent-framework / extentreports-cucumber4-adapter

Cucumber4 Adapter for Extent Framework
http://extentreports.com/docs/versions/4/java/cucumber2.html
Apache License 2.0
39 stars 22 forks source link

Spark report pie charts are showing erroneous values for branch v1.2x for extent version 5 #60

Closed grasshopper7 closed 4 years ago

grasshopper7 commented 4 years ago
  1. Executed the runner RunCukesTest.java using default values.

This is output of cucumber test execution 8 Scenarios (8 passed) 36 Steps (36 passed) 0m0.928s

But the spark report shows 4 scenarios and 18 steps only. adapter1

  1. Modified the runner by adding the tags filter tags="@foo" so only the basic_arithmetic.feature file is executed.

This is output from cucumber 6 Scenarios (6 passed) 30 Steps (30 passed) 0m0.851s

Spark shows - 2 scenarios and 8 steps. adapter2

  1. With the runner settings from point 2, changed the first scenario of basic_arithmetic.feature to fail.

6 Scenarios (1 failed, 5 passed) 30 Steps (1 failed, 29 passed) 0m0.913s java.lang.AssertionError: expected:<8.0> but was:<9.0>

Spark shows - adapter3

  1. With the runner settings from point 2, changed the first row of scenariooutline of basic_arithmetic.feature to fail.

6 Scenarios (1 failed, 5 passed) 30 Steps (1 failed, 29 passed) 0m0.775s java.lang.AssertionError: expected:<9.0> but was:<8.0>

Spark shows - adapter4

anshooarora commented 4 years ago

@grasshopper7

Thanks, checking now..

anshooarora commented 4 years ago

@grasshopper7

https://github.com/extent-framework/extentreports-java/commit/d8bcda986c23cac82aaf9446f20a40646107bf7c

This fixes the counts for scenarios and increases the count of steps. However, the steps counts do not match due to this report counting every step (including hooks) as a step.

grasshopper7 commented 4 years ago

Now it is working for the earlier cases except for the steps count. This was the behavior with version extent 4 too.

  1. Plugin does not generate the Spark report but a file named 'SparkReport' when the folder structure is not already present.

  2. Got pie chart without any features, twice when running multiple times. But unable to determine exact sequence. adapter5

Not sure of the existing step count code. To exclude hooks from step count, maybe the description variable of ExtentTest can be set to a custom string which can be used to determine that this is a hook. Currently this is set to null for hook extenttest creation

Or create a class HookExtentTest which extends ExtentTest which can be used as a filter later on. Then a new method in ExtentTest to instantiate this class.

grasshopper7 commented 4 years ago

Updated adapter code in a PR, to add duration value for each Feature in the tests tab of report. https://github.com/extent-framework/extentreports-cucumber4-adapter/pull/61

anshooarora commented 4 years ago

@grasshopper7

Thanks for your research. I will also have a look, this does look strange to me - especially since the other 2 charts did generate. Do you still have the report file? I can investigate the js that may have caused it.

The default name is SparkReport because of this config. However, the plugin should pickup the user's config file first so the report will be stored in whichever location they supply.

Regarding the steps, I will provide a way to indicate any NOP test, which will not be counted towards stats. This is a pretty significant addition as all the other reporters (especially Klov) will need to implement this as well.

grasshopper7 commented 4 years ago

Have this setting extent.reporter.spark.out=test-output/SparkReport/ in extent.properties.

-- No test-output folder - Get a file named 'SparkReport' in 'test-output' folder. If I add a html extension to this, it displays the report. -- Only test-output folder - Same as above. -- Exists test-output/SparkReport/ folder structure - Creates 'Index.html' report in SparkReport folder. As expected. -- No test-output folder AND changed extent.reporter.spark.out=test-output/SparkReport/Spark.html - Creates 'Spark.html' report in SparkReport folder.

When the folder structure is missing, the last folder seems to be used as report name.

Regarding the missing feature pie chart, encountered it again today. This possibly seems to be a display issue in my Firefox. When the report was opened, feature pie was missing but when I did a Ctrl-F5 it was displayed. Anyways I have added the report and images below. report.zip

Is the timeline part deprecated from Spark report? It is not present in the new report.

Report is not displaying docstring contents.

Scenario: Doc String Given the doc string is """ Hello there how r u?
Doing great. Whats new?
Nothing much. """

adapter7

Is it possible to add a method in MarkupHelper class which formats the docstring and replaces the line breaks with BR tags?

anshooarora commented 4 years ago

@grasshopper7

  1. Fixed the issue with file not saving in the correct location
  2. Charts: I used Brave, Edge and Chrome - unable to reproduce this on any of the 3, however I did not test with Firefox..
  3. Doc Strings - working on it
anshooarora commented 4 years ago

The following change https://github.com/anshooarora/extentreports5-cucumber5-adapter/commit/2c33272d6867a4aec31aaa1e32bd130b1249354d creates DocStrings as <pre> formatted blocks:

image

Is this the desired behavior?

grasshopper7 commented 4 years ago

Yeah that works. Did not realize MarkupHelper.createCodeBlock would work for this. Will update the existing cucumber 5 and 6 adapter code. Thanks.

grasshopper7 commented 4 years ago

I tried integrating the plugin with a cucumber V4 project and ran into Freemarker error. Using extentreportV5 master branch & cucumber4-adapterV1.2.x branch.

Below is the extent.properties settings extent.reporter.spark.start=true extent.reporter.spark.config=src/test/resources/extent-config.xml extent.reporter.spark.out=test-output/SparkReport/ screenshot.dir=test-output/ screenshot.rel.path=../

Adding the extent-config file. extent-config.xml.txt

Test runs successfully but no report is generated. Below is the error.

freemarker.log._JULLoggerFactory$JULLogger error SEVERE: Error executing FreeMarker template FreeMarker template error: The following has evaluated to null or missing: ==> config.theme [in template "commons/commons-variables.ftl" at line 3, column 9]

Complete stack is attached

stacktrace.txt

The error is coming from this line theme=config.theme?lower_case from "commons/commons-variables.ftl".

When i comment out the extent.reporter.spark.config setting, the report is generated successfully with default display settings.

anshooarora commented 4 years ago

I think it is due to the enum value not correctly mapped due to lower case word. Here is the default v5 config: https://github.com/extent-framework/extentreports-java/blob/master/config/spark-config.xml.

Will look into this..

grasshopper7 commented 4 years ago

That worked. Thanks. Any chance it can be handled at code level.

anshooarora commented 4 years ago

@grasshopper7

Fixed, latest SNAPSHOT deployed.

grasshopper7 commented 4 years ago

The following change anshooarora/extentreports5-cucumber5-adapter@2c33272 creates DocStrings as <pre> formatted blocks:

image

Is this the desired behavior?

Hi @anshooarora I added this code to the existing adapter and am getting an extra tab in the beginning and an extra new line at the end. MarkupHelper.createCodeBlock(((DocStringArgument) argument).getContent())

tagdocstr

Any chance you can have a look into this. Thanks.

anshooarora commented 4 years ago

@grasshopper7 I ran your latest code with the above addition and this is how the DocString appears for me:

image

Also attaching the HTML file:

Index.zip

anshooarora commented 4 years ago

Unrelated, but - we would need to add the KlovReporter section back to properties as its near completion and ready for release.

https://github.com/extent-framework/klov-server

Will update you soon.

anshooarora commented 4 years ago

Added JSONFormatter for v4 adapter: https://github.com/extent-framework/extentreports-cucumber4-adapter/commit/dd6fce7ab6b7ee6611063808ad231ad0294495a5

grasshopper7 commented 4 years ago

@grasshopper7 I ran your latest code with the above addition and this is how the DocString appears for me:

image

Also attaching the HTML file:

Index.zip

This is interesting. I will run this again and check. Thanks.

grasshopper7 commented 4 years ago

Added JSONFormatter for v4 adapter: dd6fce7

Will update the version 5 and 6 adapters.

I am not familiar with the JSONformatter report and have a query. This generates a json file, so in that case is there any need for a config file property (ie. location of extent-config.properties)?

anshooarora commented 4 years ago

@grasshopper7

Do you mean this: https://github.com/extent-framework/extentreports-cucumber4-adapter/blob/v1.2.x/config/extent.properties?

By default, this won't be required: extent.reporter.json.config=. But, I am keeping it here to be future-proof.

grasshopper7 commented 4 years ago

@grasshopper7 I ran your latest code with the above addition and this is how the DocString appears for me: image Also attaching the HTML file: Index.zip

This is interesting. I will run this again and check. Thanks.

The extra spaces comes up in extent version 4. I have created a new issue to make it easier to follow. - https://github.com/extent-framework/extentreports-java/issues/214

Thanks