search

webdriverio-boneyard / wdio-cucumber-framework

A WebdriverIO v4 plugin. Adapter for Cucumber testing framework.
MIT License
77 stars 61 forks source link

Include access to tags and examples in custom reporter #74

Closed christian-bromann closed 7 years ago

christian-bromann commented 7 years ago

From @cnatis on August 29, 2017 19:14

The problem

I need to create a custom reporter to integrate WDIO test results with xray (a Jira plugin). My features and scenarios are tagged with IDs and my scenarios include examples, I want to be able to access the tags on a given spec as well as either the current example being used for a test and or the index and all the examples available for that test/scenario.

TLDR; I need a way to link a scenario result to an ID preferably using a tag on the scenario or feature I need a way to report failure/success for each example of a scenario

Environment

Details

The suites, specs, and tests do not include tag or example info.

Example feature file ` @BLAH-954 Feature: Logging in

As a user I want to be able to login so that I can view my data

Background: Given I am on the login page

Login with invalid user

@BLAH-954 Scenario Outline: Automate Login with invalid user When I log in with user "" And a password "" Then the login page displays error ""

Examples: Invalid
  |username|password|error|
  |admin   |password123|Incorrect username or password|

`

Copied from original issue: webdriverio/webdriverio#2261

cnatis commented 7 years ago

I took a look at the reporter code and it looks like we already have access to at least the tags, not sure about the examples. Why isn't this info emitted in the events?

christian-bromann commented 7 years ago

@cnatis this has to be cucumber specific. WebdriverIO just runs your features in a cucumber environment. We don't implement anything Cucumber related. Also I am not an expert in Cucumber maybe if you look into the code of the wdio-cucumber-framework you will see a solution.

cnatis commented 7 years ago

@christian-bromann im going to probably fork and open a PR to at least expose the tags, ill look into whats needed to get the example data after. What I am referencing is in the code below, we have the tag data on the featureOrScenario object (comes in from an event), that object might include the example data too, ill have to look more, but I was curious why the event object is never included in the events emitted by the reporter?

https://github.com/webdriverio/wdio-cucumber-framework/blob/master/lib/reporter.js

getTitle (featureOrScenario) {
        const name = featureOrScenario.name
        const tags = featureOrScenario.tags
        if (!this.tagsInTitle || !tags.length) return name
        return `${tags.map(tag => tag.name).join(', ')}: ${name}`
    }

For example why don't these methods include the original event data in the event that is emitted to my custom reporter? If there is a specific structure that could be why, if thats the case I need more details so I can make sure my changes will be acceptable.


handleAfterScenario (event, callback) {
        const scenario = event
        this.emit('suite:end', {
            uid: this.getUniqueIdentifier(scenario),
            title: scenario.name,
            parent: this.getUniqueIdentifier(this.runningFeature),
            type: 'suite',
            file: this.getUriOf(scenario),
            duration: new Date() - this.scenarioStart
        })

        process.nextTick(callback)
    }

    handleAfterFeature (event, callback) {
        const feature = event
        this.emit('suite:end', {
            uid: this.getUniqueIdentifier(feature),
            title: feature.name,
            type: 'suite',
            file: this.getUriOf(feature),
            duration: new Date() - this.featureStart
        })

        process.nextTick(callback)
    }
christian-bromann commented 7 years ago

event object is never included in the events emitted by the reporter?

Two reasons: a. performance wise it is unwise to push to much data to the parent process (even though I don't have any proof that doing so actually degrades the performance) b. all framework adapters need to emit a similar data structure so that all reporters work across different frameworks

I wouldn't mind to transport more data to the reporter in order to provide reporters with better data but we need to make sure that we actually leverage from this. If none of the "official" WebdriverIO reporter use the data I don't see the reason why do add this.

cnatis commented 7 years ago

@christian-bromann I didnt get around to dealing with the examples just yet, I believe that will require modification of cucumber.js. However for now I have exposed the tags as I think they are important information for reporters to use.

https://github.com/webdriverio/wdio-cucumber-framework/pull/75

christian-bromann commented 7 years ago

Let's close this in favor of #75