search

webdriverio-boneyard / wdio-cucumber-framework

A WebdriverIO v4 plugin. Adapter for Cucumber testing framework.
MIT License
77 stars 61 forks source link

Wdio cucumber runner doesn't always show test results #38

Closed christian-bromann closed 7 years ago

christian-bromann commented 8 years ago

From @gsccheng on September 24, 2016 23:10

The problem

I'm not sure where to post this issue but I'll start here at the top level since I use the wdio test runner with Cucumber.

I think I may be getting a race condition where my spec reporter doesn't always show the test result because the runner is ending prematurely. It seems to only occur when there are multiple browser commands, whether they are in the same step definition or not.

Environment

I simplified my code into these code snippets. I'm wondering if anyone can reproduce this.

Here's my gist to the config file

// features/auth.feature
 Feature: Auth
   Some description

  Scenario: Google Auth
   Given this should work
// features/step_definitions/auth.js
import { expect } from 'chai';
[...]
  this.Given(/^this should work$/, function () {
    browser.url('https://google.com');
    browser.getSource();
    expect('foo').to.exist;
  });
// Output: 
myApp [testing] :> npm run ui-test

> react-redux-universal-hot-example@0.9.0 ui-test /Users/GabbAHH/Desktop/Dev/myApp
> better-npm-run ui-test

running better-npm-run in /Users/GabbAHH/Desktop/Dev/myApp
Executing script: ui-test

to be executed: ./node_modules/.bin/wdio wdio.conf.js

=======================================================================================
Selenium 2.0 / webdriver protocol bindings implementation with helper commands in nodejs.
For a complete list of commands, visit http://webdriver.io/api.html.
=======================================================================================

[16:02:15]  COMMAND POST     "/wd/hub/session"
[16:02:15]  DATA        {"desiredCapabilities":{"javascriptEnabled":true,"locationContextEnabled":true,"handlesAlerts":true,"rotatable":true,"maxInstances":6,"browserName":"chrome","loggingPrefs":{"browser":"ALL","driver":"ALL"},"requestOrigins":{"url":"http://webdriver.io","version":"4.2.8","name":"webdriverio"}}}
[16:02:17]  INFO    SET SESSION ID 42e1e89c-0bbb-4553-b7d7-bce852f847cf
[16:02:17]  RESULT      {"applicationCacheEnabled":false,"rotatable":false,"mobileEmulationEnabled":false,"networkConnectionEnabled":false,"chrome":{"chromedriverVersion":"2.23.409710 (0c4084804897ac45b5ff65a690ec6583b97225c0)","userDataDir":"/var/folders/59/6pd62vqn20j5qx2hv1l1w9_r0000gn/T/.org.chromium.Chromium.FUqeyq"},"takesHeapSnapshot":true,"pageLoadStrategy":"normal","databaseEnabled":false,"handlesAlerts":true,"hasTouchScreen":false,"version":"53.0.2785.116","platform":"MAC","browserConnectionEnabled":false,"nativeEvents":true,"acceptSslCerts":true,"webdriver.remote.sessionid":"42e1e89c-0bbb-4553-b7d7-bce852f847cf","locationContextEnabled":true,"webStorageEnabled":true,"browserName":"chrome","takesScreenshot":true,"javascriptEnabled":true,"cssSelectorsEnabled":true}
[16:02:18]  COMMAND POST     "/wd/hub/session/42e1e89c-0bbb-4553-b7d7-bce852f847cf/url"
[16:02:18]  DATA        {"url":"https://google.com"}
[16:02:25]  RESULT      null
[16:02:25]  COMMAND GET      "/wd/hub/session/42e1e89c-0bbb-4553-b7d7-bce852f847cf/source"
[16:02:25]  DATA        {}
[16:02:25]  RESULT      "<!DOCTYPE html><html xmlns=\"http://www.w3.org/1999/xhtml\" itemscope=\"\" itemtype=\"http://schema.org/WebPage\" lang=\"en\"><head><meta content=\"Search the world's information, including webpages, ... (230174 more bytes)
[16:02:25]  COMMAND DELETE   "/wd/hub/session/42e1e89c-0bbb-4553-b7d7-bce852f847cf"
[16:02:25]  DATA        {}
[16:02:25]  RESULT      null

Now if I remove the second command of browser.getSource(), I am able to complete the test properly: So, 

// features/step_definitions/auth.js
  this.Given(/^this should work$/, function () {
    browser.url('https://google.com');
    expect('foo').to.exist;
  });

gives: 

myApp [testing] :> npm run ui-test

> react-redux-universal-hot-example@0.9.0 ui-test /Users/GabbAHH/Desktop/Dev/myApp
> better-npm-run ui-test

running better-npm-run in /Users/GabbAHH/Desktop/Dev/myApp
Executing script: ui-test

to be executed: ./node_modules/.bin/wdio wdio.conf.js

=======================================================================================
Selenium 2.0 / webdriver protocol bindings implementation with helper commands in nodejs.
For a complete list of commands, visit http://webdriver.io/api.html.
=======================================================================================

[16:02:41]  COMMAND POST     "/wd/hub/session"
[16:02:41]  DATA        {"desiredCapabilities":{"javascriptEnabled":true,"locationContextEnabled":true,"handlesAlerts":true,"rotatable":true,"maxInstances":6,"browserName":"chrome","loggingPrefs":{"browser":"ALL","driver":"ALL"},"requestOrigins":{"url":"http://webdriver.io","version":"4.2.8","name":"webdriverio"}}}
[16:02:44]  INFO    SET SESSION ID 2ef670fe-d842-4052-8d24-260e48fcb376
[16:02:44]  RESULT      {"applicationCacheEnabled":false,"rotatable":false,"mobileEmulationEnabled":false,"networkConnectionEnabled":false,"chrome":{"chromedriverVersion":"2.23.409710 (0c4084804897ac45b5ff65a690ec6583b97225c0)","userDataDir":"/var/folders/59/6pd62vqn20j5qx2hv1l1w9_r0000gn/T/.org.chromium.Chromium.GidKiN"},"takesHeapSnapshot":true,"pageLoadStrategy":"normal","databaseEnabled":false,"handlesAlerts":true,"hasTouchScreen":false,"version":"53.0.2785.116","platform":"MAC","browserConnectionEnabled":false,"nativeEvents":true,"acceptSslCerts":true,"webdriver.remote.sessionid":"2ef670fe-d842-4052-8d24-260e48fcb376","locationContextEnabled":true,"webStorageEnabled":true,"browserName":"chrome","takesScreenshot":true,"javascriptEnabled":true,"cssSelectorsEnabled":true}
[16:02:45]  COMMAND POST     "/wd/hub/session/2ef670fe-d842-4052-8d24-260e48fcb376/url"
[16:02:45]  DATA        {"url":"https://google.com"}
[16:02:50]  RESULT      null
[16:02:50]  COMMAND DELETE   "/wd/hub/session/2ef670fe-d842-4052-8d24-260e48fcb376"
[16:02:50]  DATA        {}
[16:02:50]  RESULT      null
------------------------------------------------------------------
[chrome #0a] Session ID: 2ef670fe-d842-4052-8d24-260e48fcb376
[chrome #0a] Spec: /Users/GabbAHH/Desktop/Dev/myApp/features/auth.feature
[chrome #0a] Running: chrome
[chrome #0a]
[chrome #0a]   Auth
[chrome #0a]
[chrome #0a]     Google Auth
[chrome #0a]         ✓ this should work
[chrome #0a]
[chrome #0a]
[chrome #0a] 1 passing (9s)
[chrome #0a]

Copied from original issue: webdriverio/webdriverio#1611

christian-bromann commented 8 years ago

From @msegado on October 6, 2016 22:13

@gsccheng, do you have screenshots enabled? I'm wondering if this is the same underlying issue as #1591.

gsccheng commented 8 years ago

Thanks for addressing my issue.

Following #1591 I've commented out screenshotPath and also tried setting it to a nonexistent path and either way still produces the same problem the majority of the time. Yes, in a minority of the time I would get the report as expected so the problem doesn't occur all the time.

wvankuipers commented 7 years ago

I was looking into this but I have a hard time reproducing this. @gsccheng is this still an issue? If so could you try this code and share the output?

// features/step_definitions/auth.js
this.Given(/^this should work$/, function () {
    try {
            browser.url('https://google.com');
            var source = browser.getSource();

            expect('foo').to.exist;
        } catch (e) {
            console.log('Error: ', e);
            throw e;
        }
  });
gsccheng commented 7 years ago

Thanks for looking into this. Since I've had this problem, I ended up switching to a new computer, and updating all the relevant dependencies. I'm not sure what caused it, but I've just finished all my UI-tests without any problems. This can be closed.

christian-bromann commented 7 years ago

@gsccheng glad everything worked out.