Execution Summary details at top right corner

behave-contrib / behave-html-formatter

HTML formatter for Behave

https://pypi.org/project/behave-html-formatter/

GNU General Public License v3.0

7 stars 10 forks source link

Execution Summary details at top right corner #29

Open jega-kathir opened 2 years ago

jega-kathir commented 2 years ago

Hi I ran and created one sample html report. I ran for 2 features which has 11 scenarios. In this, 4 scenarios are failed (2 from Feature 1 and 2 from feature 2) But report says:

Features: failed: 2
Scenarios: failed: 2
Steps: untested: 8

Here am not able to understand how the "Scenario : failed" and "Steps : Untested" got calculated.

bittner commented 2 years ago

Do you have some code we can look at to better understand the problem?

What is the console output of your tests when you don't use the HTML formatter?

jega-kathir commented 2 years ago

My console Output: 0 features passed, 2 failed, 0 skipped 7 scenarios passed, 4 failed, 0 skipped 40 steps passed, 4 failed, 0 skipped, 0 undefined Took 0m36.643s

But in report html error.docx

Features: failed: 2
Scenarios: failed: 2
Steps: untested: 8
Finished in 36.6 seconds

jenisys commented 2 years ago

@jega-kathir The untested state is the initial state before the test run. Therefore, if you see untested steps this means that these steps did not get executed during the test run.

EXAMPLES:

If the second step in a scenario fails the remaining steps in this scenario have still the state untested (because the execution normally stops after the first failed step in a scenario).
If you are using dry-run mode, all steps are untested (unless you have undefined steps that have no step implementation) because they do not get executed during the test run.
If you want to stop the test run after the first failure, the other scenarios, that would normally be executed, are untested.

OTHERWISE: Your console output example above is weird. untested steps are optional in the summary. Therefore, untested steps are only shown if some exist.

lukjak commented 1 year ago

I have a similar issue.

Behave output states:

0 features passed, 3 failed, 0 skipped
5 scenarios passed, 7 failed, 0 skipped
11 steps passed, 7 failed, 0 skipped, 0 undefined
Took 9m40.783s

And HTML report (with v 0.9.10) shows:

Features: failed: 3
Scenarios: failed: 4, passed: 1
Steps: passed: 3, failed: 3, untested: 3
Finished in 580.8 seconds

lukjak commented 1 year ago

Re my previous comment. Feature data are OK. Scenario data value accumulate all scenario outline executions into single item, while behave treats each execution as a separate scenario. I have no explanation for steps value - it seems to be completely wrong.

Here is results outline:

Feature 1
Scenario 11
Step A - pass
Step B - fail
Scenario 12
Step A - pass
Step C - fail

Feature 2
Scenario Outline 21@1
Step D - pass
Step E - pass
Scenario Outline 21@2
Step D - pass
Step E - pass
Scenario Outline 21@3
Step D - pass
Step E - pass
Scenario Outline 22@1
Step F - pass
Scenario Outline 22@2
Step F - pass
Scenario Outline 22@3
Step F - fail
Scenario Outline 22@4
Step F - fail
Scenario Outline 22@5
Step F - fail
Scenario Outline 22@6
Step F - fail

Feature 3
Scenario 31
Step G - pass
Step H - fail