Feature: Export Scenario Definitions

[helmutchecker]

The problem When working with other Test Management Tools like Jira and Xray, there might be need to keep Scenario Definitions in Sync. As it is probably not working out to maintain scenarios in Jira itself, it would be good to sync the latest definition from the repository to Jira to have traceability to the requirements and as well an overview over the implementation.

A solution Manually copying and updating scenario definitions is obviously no solution.

If there was a way to export scenario definitions, the output could be used to connect to an API of a Test Management system or another tool to update test cases there. As there are several CLI commands already existing like bddgen export which exports only the steps , a similar command could export the full scenario definition.

[vitalets]

Could you show an example, what is the output for scenario definitions export? Isn't it just feature files itself?

[NikkTod]

Yes, I think he means the feature files and the steps inside to be able to push them into external Test Management system.

[vitalets]

Understood, in that case we need to know the format. Even if export as json, it's helpful to see an example beforehand.

[NikkTod]

Hi @vitalets

I had past project where we used Azure Devops and Playwright with Specflow (C#). We had to send the bdd steps (also trace attachments for failed scenarious) to Azure Test Case item and we used their API to do that. So usually test management applications have ID object, which we can use in the API call to update the item (in our case testcase item). That ID we can put as tag on the bbd scenario.

So we have the following flow:

1. We create a test case within the Test Management system (could be done manually or via API)
2. Upon creation this test case with have now an ID
3. We put that ID as a tag on a particular BDD Scenario
4. We can now use that ID and via the test management API and set/update all relevant item fields: test item title <-> bdd scenario title test item step <-> bdd scenario step test item attachment <-> e.g. playwright trace file test item status <-> scenario status (pass, fail, skip)

So with Specflow or Cucumber we have possibility to get the following data about the scenario currently being executed: Feature: Title/Tag Scenario: Title/All steps/Tag/Error/Status Scenario Step: Executed bdd step (including the data table value), status (result of the executed step). Do we have @AfterStep and @BeforeStep capability?

Specflow -> https://docs.specflow.org/projects/specflow/en/latest/Bindings/ScenarioContext.html#scenariocontext-scenarioinfo

Cucumber -> cannnot find the documentation, but as an example to get scenario title you can do

Before((scenario)=> {
      console.log('scenario.pickle.name);
    });

I think all that data comes as string or array of strings and then everyone can do whatever they like with that information. Sending it to Test Management systems (Xray, Jira, AzureDevops, Test Rail) or do custom reports.

This ticket is very needed as people will have single source of truth for their test cases. So if you change some bdd scenario step/steps in your code and you run the test those steps will be sent via the test management api and will updated that test case, which resides in the test management system.

[vitalets]

Thanks you @NikkTod , that's very helpful! I thought of this ticket as a static export of features info without test run. From your examples I see that execution statuses are also required. Isn't it just a reporting task? For example, if I use playwright-azure-reporter, it sends test execution data with feature info and testIds. Or I miss something?

[NikkTod]

@vitalets It`s not just a reporting task, it is more of a way to contol all your test details and run statuses from one place (our code repo), so that you are able to push them in a test management system that your team uses for other non-technical people to see (manual testers/stakeholders). These test management systems we use to organize all our test procesess, not only reporting.

The azure-reporter is limited, yes there is similarity in the way that it also takes data, probably from the created report and insert it in AzureDevOps, but is a framework working only with Azure DevOps.

I think that by providing us the ability to get the above mentioned data (steps, tags, statuses) at runtime will allow us to use any Test Management tool as all of them have own API.

Many teams are using different tools to create their testcases/ test suites and manage their sprint cycles and reports.

Let me give some example screenshots:

TestRail:

TestCase details

Test Cases overview

Xray

So, all in all, if lets say I modify the title and my bdd steps of a test scenario and then run the test within my VSCode, after test execution I want to be able to send data such as steps, title, tags, status via API to update the test management system.

That way if a user (manual tester/ stackholder) opens that testcase in the test management system he will be able to see newly updated title, the new steps that I have executed and of course the status of the test case would be set (pass/fail).

[vitalets]

The flow and benefits are now clear. To me it sounds like standard Cucumber json report covers all these needs. I've executed the following feature with that reporter:

Feature: Playwright Home Page

    @testId:12345
    Scenario: Check title
        Given I am on Playwright home page
        When I click link "Get started"
        Then I see in title "Installation"

Output json contains info about feature, tags and all steps with execution status. It can be sent to the third party API to update test cases, using testId from tags:

report.json

```json [ { "id": "chromium-›-playwright-home-page", "line": 1, "keyword": "Feature", "name": "Playwright Home Page", "tags": [], "uri": "features/homepage.feature", "description": "", "elements": [ { "id": "playwright-home-page;check-title", "keyword": "Scenario", "type": "scenario", "name": "Check title", "tags": [{ "name": "@testId:12345" }], "description": "", "line": 4, "steps": [ { "arguments": [], "keyword": "Given ", "line": 5, "name": "I am on Playwright home page", "result": { "status": "passed", "duration": 712000000 } }, { "arguments": [], "keyword": "When ", "line": 6, "name": "I click link \"Get started\"", "result": { "status": "passed", "duration": 91000000 } }, { "arguments": [], "keyword": "Then ", "line": 7, "name": "I see in title \"Installation\"", "result": { "status": "passed", "duration": 164000000 } } ] } ] } ] ```

Does it miss something what you need in TestRail / XRay?

[NikkTod]

Thansk @vitalets. That would work yes, report have a lot of data that we can use. It will require a bit more code in order to parse the json and extract the needed data (it can have hundrets of test inside).

Should be something like look first for tag @testId:12345 and then go up and down to extract feature name, scenario name and steps.

[vitalets]

Yes, that json should be adopted to the target system. But maybe there are existing integrations, because Cucumber json is not something new.

So, the solution for the original request - output Cucumber json / junit reporters and upload them to the target system. The configuration in Playwright config is the following:

export default defineConfig({
  // ...
  reporter: [
    cucumberReporter("json", { outputFile: "cucumber-report/report.json" }),
    cucumberReporter("junit", { outputFile: "cucumber-report/report.xml" }),
  ],
});

Specifically, for TestRail I've found this guide in their integration docs.

Feel free to re-open this issue, better with concrete system use-case. @NikkTod thanks for providing the context!