cucumber / cucumber-jvm

Cucumber for the JVM
https://cucumber.io
MIT License
2.69k stars 2.02k forks source link

Use message based TestNG formatter #2863

Closed mpkorstanje closed 3 months ago

mpkorstanje commented 3 months ago

🤔 What's changed?

Replace the TestNGFormatter with the message based implementation from cucumber-junit-xml-formatter. This achieves several goals:

This results in several changes/improvements:

  1. Use the long naming strategy to include the scenario name, examples group and example numbers.
Feature: Examples Tables
  Scenario Outline: Eating cucumbers
    Given there are <start> cucumbers
    When I eat <eat> cucumbers
    Then I should have <left> cucumbers
    Examples: These are passing
      | start | eat | left |
      |    12 |   5 |    7 |
      |    20 |   5 |   15 |
    Examples: These are failing
      | start | eat | left |
      |    12 |  20 |    0 |
      |     0 |   1 |    0 |

This feature previously contained the examples

 - Eating cucumbers
 - Eating cucumbers_2
 - Eating cucumbers_3
 - Eating cucumbers_4

And will now contain the examples:

 - Examples Tables - Eating cucumbers - These are passing - Example #1.1
 - Examples Tables - Eating cucumbers - These are passing - Example #1.2
 - Examples Tables - Eating cucumbers - These are failing - Example #2.1
 - Examples Tables - Eating cucumbers - These are failing - Example #2.2
  1. Rename the suite from io.cucumber.core.plugin.TestNGFormatter to Cucumber
<suite name="Cucumber" duration-ms="61">
  <test name="Cucumber" duration-ms="61">
  1. When a scenario is skipped or failed the executed steps are always included in the message element.
<test-method name="Eating cucumbers - These are failing - Example #2.1" status="FAIL" duration-ms="7" started-at="1970-01-01T00:00:00.017Z" finished-at="1970-01-01T00:00:00.024Z">
  <exception class="AssertionError">
    <message><![CDATA[
Given there are 12 cucumbers................................................passed
When I eat 20 cucumbers.....................................................passed
Then I should have 0 cucumbers..............................................failed
]]></message>
  </exception>
</test-method>
<test-method name="Skipping from a step causes the rest of the scenario to be skipped" status="SKIP" duration-ms="5" started-at="1970-01-01T00:00:00.013Z" finished-at="1970-01-01T00:00:00.018Z">
  <exception class="The scenario has skipped step(s)">
    <message><![CDATA[
Given I skip a step.........................................................skipped
And a step that is skipped..................................................skipped
]]></message>
    <full-stacktrace>
      <![CDATA[The scenario has skipped step(s)]]>
    </full-stacktrace>
  </exception>
</test-method>
  1. The empty scenario and empty report are considered passing

To be consistent with the exit code of the Cucumber process empty scenarios and test runs are considered passing.

🏷️ What kind of change is this?

📋 Checklist: