mucsi96 / nightwatch-cucumber

[DEPRECATED] Cucumber.js plugin for Nightwatch.js.
http://mucsi96.github.io/nightwatch-cucumber
MIT License
236 stars 73 forks source link

Get an unexpected "Error: function timed out after" #244

Closed GRme-zz closed 7 years ago

GRme-zz commented 7 years ago

I'm submitting a ... (check one with "x")

[x] bug report
[ ] feature request
[ ] support request => Please do not submit support request here, instead use https://groups.google.com/d/forum/nightwatch-cucumber

Current behavior I work with Nightwatch-Cucumber and the PageObject Pattern and I get an unexprected "Error: function timed out after 60000 milliseconds".

Expected/desired behavior All Nightwatch-Cucumber checks (like visibility checks) have to fail and no timeout problem has to occur.

Reproduction of the problem As pre-conditions I set the default timeout (60 seconds) globally in timeout.js:

const {defineSupportCode} = require('cucumber');

defineSupportCode(({setDefaultTimeout}) => {
  setDefaultTimeout(60 * 1000);
});

...and I set the waitForConditionTimeout and waitForConditionPollInterval for Nightwatch in nightwatch.conf.js:

  test_settings: {
    default: {
      globals : {
        "waitForConditionTimeout": 30000,
        "waitForConditionPollInterval": 500
      },

Now I have a Cucumber test that has to fail. So, I want to test the right behaviour of the testframework:

Feature: only a test feature

  Scenario: only a test Scenario
    #first step should pass
    Given a user is on a details page with id "123"
    #second step should fail
    Then user is on the first page of the booking funnel

And here are the two step definitions:

const {client} = require('nightwatch-cucumber');
const {defineSupportCode} = require('cucumber');

const detailsPage = client.page.detailsPageView();
const bookingPage = client.page.bookingStepOnePageView();

defineSupportCode(({Given, When, Then}) => {

  Given(/^a user is on a details page with id "([^"]*)"$/, (id) => {
    return detailsPage.openUrlWithId(client, id);
  });

  Then(/^user is on the first page of the booking funnel$/, () => {
    return bookingPage.checkFirstStepOfBookingFunnel(client);
  });
});

And here is the page object function for first cucumber step (detailsPageView.js):

module.exports = {
  elements: {},
  commands: [{
    openUrlWithId(client, id) {
      return client
        .url('http://test.com?id=' + id);
    }
  }]
};

...and the page object function for second cucumber step (bookingStepOnePageView):

const offerSummary = 'div[id="offerSummary"]';

module.exports = {
  elements: {},
  commands: [{
    checkFirstStepOfBookingFunnel(client) {
      client.expect.element(offerSummary).to.be.visible.after();
      return client;
    },
  }]
};

Now if I will run my test I expect that the second cucumber step will fail, because the first page of booking funnel is not loaded and present. So, the visibility check in booking page object function client.expect.element(offerSummary).to.be.visible.after(); has to fail. Now I expect that the defined "waitForConditionTimeout":30000 in nightwatch.conf.js will use in that case and the visibility check will fail after 30 seconds, but I get a timeout error after 60 seconds how it is defined in timeout.js with setDefaultTimeout(60*1000).

And additionally my test run (test process via nightwatch --env chrome) doesn't end and the browser window doesn't close. So I have to end the run (process) manually with ctrl + c.

Here you can see the output:

grme:e2e-web-tests GRme$ npm run test-chrome

> e2e-web-tests@0.0.2 test-chrome /Users/GRme/projects/myProject/e2e-web-tests
> nightwatch --env chrome

Starting selenium server... started - PID:  29642
Feature: only a test feature

  @run
  Scenario: only a test Scenario
  ✔ Given a user is on a details page with id "123"
  ✖ Then user is on the first page of the booking funnel

Failures:

1) Scenario: only a test Scenario - features/testFeature.feature:4
   Step: Then user is on the first page of the booking funnel - features/testFeature.feature:6
   Step Definition: features/step_definitions/bookingFunnelStepDefinition.js:33
   Message:
     Error: function timed out after 60000 milliseconds
         at Timeout._onTimeout (/Users/GRme/projects/myProject/e2e-web-tests/node_modules/cucumber/lib/user_code_runner.js:91:22)
         at ontimeout (timers.js:488:11)
         at tryOnTimeout (timers.js:323:5)
         at Timer.listOnTimeout (timers.js:283:5)

1 scenario (1 failed)
2 steps (1 failed, 1 passed)
1m09.648s
^C
grme:e2e-web-tests GRme$

On last line you can see the ^C as my manually stopped test process.

Especially when I want to execute a test suite with maybe two cucumber tests. The first test is the one I explained and the second is a one where I expect a pass. In this case both tests will fail for me, because in the second test the visibility check of the first test (client.expect.element(offerSummary).to.be.visible.after();) will fail and I don't know why.

This is my console output for both tests (the second one has to pass!):

grme:e2e-web-tests GRme$ npm run test-chrome

> e2e-web-tests@0.0.2 test-chrome /Users/GRme/projects/myProject/e2e-web-tests
> nightwatch --env chrome

Starting selenium server... started - PID:  29691
Feature: only a test feature

  @run
  Scenario: only a test Scenario
  ✔ Given a user is on a details page with id "123"
  ✖ Then user is on the first page of the booking funnel

  @run
  Scenario: only a test Scenario 2
  ✖ Given a user is on a details page with id "123"

Failures:

1) Scenario: only a test Scenario - features/testFeature.feature:4
   Step: Then user is on the first page of the booking funnel - features/testFeature.feature:6
   Step Definition: features/step_definitions/bookingFunnelStepDefinition.js:33
   Message:
     Error: function timed out after 60000 milliseconds
         at Timeout._onTimeout (/Users/GRme/projects/myProject/e2e-web-tests/node_modules/cucumber/lib/user_code_runner.js:91:22)
         at ontimeout (timers.js:488:11)
         at tryOnTimeout (timers.js:323:5)
         at Timer.listOnTimeout (timers.js:283:5)

2) Scenario: only a test Scenario 2 - features/testFeature.feature:9
   Step: Given a user is on a details page with id "123" - features/testFeature.feature:10
   Step Definition: features/step_definitions/detailStepDefinition.js:12
   Message:
     Expected element <div[id="offerSummary"]> to be visible - element was not found - Expected "visible" but got: "not found"
         at Page.checkFirstStepOfBookingFunnel (/Users/GRme/projects/myProject/e2e-web-tests/pageobjects/bookingStepOnePageView.js:49:21)
         at World.Then (/Users/GRme/projects/myProject/e2e-web-tests/features/step_definitions/bookingFunnelStepDefinition.js:34:24)

2 scenarios (2 failed)
3 steps (2 failed, 1 passed)
1m11.448s
^C
grme:e2e-web-tests GRme$

What is the expected behavior? Please see Reproduction of the problem

What is the motivation / use case for changing the behavior? Maybe my tests will fail and the worst thing is that my test process doesn't end or keep going on with the next cucumber tests.

Please tell us about your environment: Mac OS X 10.12.5 Chrome Browser 59.0.3071.115 npm 5.0.4

├── cucumber@2.3.1
├── nightwatch@0.9.16
└── nightwatch-cucumber@7.1.10

v8.0.0

I hope you can help me :)

Thanks, Martin

GRme-zz commented 7 years ago

I tried debugging a little bit more. Maybe it's a bug with the Expect?

I set my global timeout to 20 seconds:

const {defineSupportCode} = require('cucumber');

defineSupportCode(({setDefaultTimeout}) => {
  setDefaultTimeout(20 * 1000);
});

And now I have a following Expect check:

client.expect.element(myElement).text.to.contain(myText).after(10000);

In my case myElement can not be found in DOM to produce an error. But now I get the error message Error: function timed out after 20000 milliseconds, but I exprect that the test fails after 10 seconds because the element myElement can not be found in DOM.

After that I tried to replace the Expect API method with Nightwatch Command:

client
  .waitForElementVisible(arrivalDepartureDate, 10000)
  .assert.containsText(myElement, myText);

And now I get the right error after 10 seconds ERROR: Unable to locate element: ".Grid__colM3___EyfpA" using: css selector.

It's a workaround to forgo Expect, in my case now I have two commands to check element and their text. With Expect I can do it both in one line command.

mucsi96 commented 7 years ago

Hi @GRme! Thanks for your feedback! Can you try to move the page object variable declaration inside the steps? Does it change anything?

const detailsPage = client.page.detailsPageView();
const bookingPage = client.page.bookingStepOnePageView();
defineSupportCode(({Given, When, Then}) => {

  Given(/^a user is on a details page with id "([^"]*)"$/, (id) => {
    const detailsPage = client.page.detailsPageView();
    return detailsPage.openUrlWithId(client, id);
  });

  Then(/^user is on the first page of the booking funnel$/, () => {
    const bookingPage = client.page.bookingStepOnePageView();
    return bookingPage.checkFirstStepOfBookingFunnel(client);
  });
});
GRme-zz commented 7 years ago

Hi @mucsi96,

sorry my delayed answer. the your problem-solving approach has unfortunately no impact.

But I have spent much more time in this bug to analyze the reason for that strange behaviour. Maybe it's a bug in Nightwatch.js, but I hope you can help me to find a solution.

I think Expect is not the really reason for it. I tried the following:

  1. I set the defaultTimeout to 20 seconds:
const {defineSupportCode} = require('cucumber');
defineSupportCode(({setDefaultTimeout}) => {
  setDefaultTimeout(20 * 1000);
});
  1. I added a method to PageObject with five waitForElementVisible() in it
checkFirstStepOfBookingFunnel(client) {
  return client
    .waitForElementVisible(element1, 5000)
    .waitForElementVisible(element2, 5000)
    .waitForElementVisible(element3, 5000)
    .waitForElementVisible(element4, 5000)
    .waitForElementVisible(element5, 5000);
},

And now I played a little bit with these five checks and the five CSS selectors of the elements:

  1. If element1 can not be found, but all other elements from element2...element5, then my test failed after exactly real time 5 seconds with the message:

Timed out while waiting for element <div.abc> to be present for 5000 milliseconds. - Expected "visible" but got: "not found"

--> That's the right behaviour!!!

  1. If element1 and element2 can not be found, but all other elements from element3..element5, then my test failed only after exactly real time 10 seconds with the message:

Timed out while waiting for element <div.abc> to be present for 5000 milliseconds. - Expected "visible" but got: "not found"

--> That's not the right behaviour, because the test have to fail after 5000ms while the waitForElement(element1, 5000), but the next waitForElementVisible(element2, 5000) seems to be executed as well.

  1. All elements element1..element5 can not be found, then my test failed only after exactly real time 20 seconds with the message:

Error: function timed out after 20000 milliseconds

--> That's also not the right behaviour, because the test have to fail after 5000ms while the waitForElement(element1, 5000), but now I get a timeout for the setDefaultTimeout(20000). In that case the following automated cucumber testcase will also crashes, because of the still running waitForElement() checks which only ends after 25 seconds (for every waitForElementVisible of the five elements).

CONCLUSION It seems to be that the waitForElement functions wait a given time in milliseconds for an element in the page, but the test does not fail after the first check fails. So, all waitForElement functions inside a PageObject function will be executed in sequence, after one another. And so the wait times, given as parameters, sum up to a total wait time. And in that case that the total wait time is larger than the defined defaultTimeout it will also have an impact on the next testcases that will be executed.

So, I hope you understand what I mean and you can reproduce this fact. Maybe you have a problem-solving approach for me :) :) :)

Thanks, Martin

GRme-zz commented 7 years ago

I've added a simple project to GitHub (https://github.com/GRme/Nightwatch-Cucumber-TestProject). Hopefully you understand the project structure.

In the PageObject pageobjects/myPageView.js I have the following function:

checkAllElementsAreVisible(client) {
  return client
    //selector 'invalidGoogleLogo' can not be found
    .waitForElementVisible(invalidGoogleLogo, 5000)
    //selector 'invalidSearchField' can not be found
    .waitForElementVisible(invalidSearchField, 5000)
    //selector 'invalidSearchButton' can not be found
    .waitForElementVisible(invalidSearchButton, 5000);
},

All the three checked element selector can not be found on google page. I've manipulated them to show the problem. So, you and me expect that the test will fail after 5 seconds for the visibility check waitForElementVisible(invalidGoogleLogo, 5000). But that's not reality. The test fails only after 15 seconds (real time, stopped with stop watch) for the visibility check waitForElementVisible(invalidSearchButton, 5000). The message says that the check failed after 5000ms, but that's not the reality.

Hopefully you can reproduce it on your machine and then you understand my problem.

Thx Martin

GRme-zz commented 7 years ago

Hi,

I've found a workaround, but it's not more than a workaround. I use Babel support with async to solve my problem:

    async checkAllElementsAreVisibleWithAsync(client) {
      //selector 'invalidGoogleLogo' can not be found
      await client.waitForElementVisible(invalidGoogleLogo, 5000);
      //selector 'invalidSearchField' can not be found
      await client.waitForElementVisible(invalidSearchField, 5000);
      //selector 'invalidSearchButton' can not be found
      await client.waitForElementVisible(invalidSearchButton, 5000);
      return client;
    },

I've added an example also to the GitHub project (https://github.com/GRme/Nightwatch-Cucumber-TestProject).

GRme-zz commented 7 years ago

@mucsi96 Is there any possibility to get a fast fix for that problem? My wotkaround with async doesn't work as stable as I want. Maybe everyone of us should have this problem when using more than one waitFor within one function.

THX Martin

mucsi96 commented 7 years ago

Hi Martin, Sorry for long delay. Currently I am on vacation. Will be back in two weeks :)

GRme-zz commented 7 years ago

First you have to enjoy your holiday 😊

mucsi96 commented 7 years ago

Hi @GRme! Can you check this behavior using pure Nightwatch? So without this package. If the issue still exists you can try to submit the issue for Nightwatch guys. Also I think they will appreciate a pull request. :)

GRme-zz commented 7 years ago

@mucsi96 I discussed this issue also with the guys of Nightwatch. They mean that's no problem with Nighwatch but with Nightwatch-Cucumber. That's the reason I hope the bug is a part of your project.

One guy tested it with native Nightwatch (without Cucumber) and it worked correct.

Here you can find my discussion with the Nightwatch guys: https://github.com/nightwatchjs/nightwatch/issues/1522

GRme-zz commented 7 years ago

@mucsi96 Is this issue still alive on your side?

mucsi96 commented 7 years ago

Yes. Will check it...

mucsi96 commented 7 years ago

Thanks! I could reproduce the issue. selenium-debug-nightwatch-cucumber.txt selenium-debug-nightwatch.txt

mucsi96 commented 7 years ago

It look its working fine in Nightwatch but as you mention not ok with Nightwatch-Cucumber. I have test your case 3. "All elements element1..element5 can not be found, then my test failed only after exactly real time 20 seconds with the message:"

mucsi96 commented 7 years ago

Fixed in https://github.com/mucsi96/nightwatch-cucumber/releases/tag/v8.0.6 .Thanks for your feedback! Thanks to You I improved and simplified the error handling a lot! 🎉