nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.79k stars 1.31k forks source link

beforeEach does not respect the done argument #1230

Closed NickStefan closed 5 years ago

NickStefan commented 7 years ago

globals:

module.exports = {
    beforeEach: function(browser, done){
           browser.timeouts('page load', 30000, function(){
                  setTimeout(function(){
                          done()
                   }, 10000);
           })
    }
};

No matter how long I set the set timeout, the browser continues on to the tests following it.

senocular commented 7 years ago

More specifically, the completion of the command queue in global beforeEach does not wait for done. You're basically limited to one or the other - using commands, or doing something async with a done() completion. As a workaround you should be able to use a perform() to do your async stuff while using commands.

module.exports = {
    beforeEach: function(browser, done){
           browser.timeouts('page load', 30000);
           browser.perform(function(performDone){
                  setTimeout(function(){
                          performDone()
                   }, 10000);
           })
    }
};

I'm actually in the process of working on a fix for this now (among other similar failures like this). Hopefully it won't be too much longer.

NickStefan commented 7 years ago

thanks. I was starting to question how both could coexist.

  1. "its javascript, so wont basically every single thing in this library (commands, assertions, page objects etc), need to be wrapped in some horrible async.js style control flow?" OR
  2. "I guess each sequential function call could be adding to a global / singleton queue. still not sure if that guarantees async problems go away".

Basically, all of our tests work locally, but they randomly hang on different test suites on the CI server. After tons of logging I was starting to doubt the async story. Thus why I've been trying to find async things that don't work here as they say they do.

It sounds like scenario 2 is what is happening under the hood?

senocular commented 7 years ago

Yeah, pretty much # 2. Nightwatch commands (most) are added to a command queue and executed in sequence asynchronously. These commands are usually wrappers for at least one, often two, WebDriver request(s) which is handled over HTTP, so each of those are inherently asynchronous on their own. Whenever you call a command from browser, you're adding the operation handling those HTTP requests (and/or possibly other things) to a queue which is traversed after the test/hook is called. Any request in the queue has to wait until the other one completes before it can be run.

I'm not sure if the command queue is described in any great detail in the documentation at all, but it probably should be (maybe in the wiki? I haven't checked it). What's more confusing is that some calls aren't added to the queue. Inherited assert commands like browser.assert.ok are not, and while custom commands are, custom page object commands aren't. So it can get confusing.

The queue itself is fairly solid. It has some quirks that custom commands have to deal with, but otherwise its pretty well off. What's not great is the implementation around calling your test and hook functions. There are ... I believe 4? different implementations on how they're called internally and how the queue and done interact with one another - each of which flawed in their own ways. That's what I'm working on fixing now.

NickStefan commented 7 years ago

Thanks for the explanation. Its helped me narrow down where our tests random failing may be coming from.

nightwatch was appearing to be the async problem, but I've figured out that the problem is actually selenium server randomly falling over. https://github.com/nightwatchjs/nightwatch/issues/1223 .

emadow commented 6 years ago

This seems to still be an issue -- very surprised this isn't blocking a whole host of folks

beatfactor commented 6 years ago

@emadow is that the case for v1.0.6 as well?

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had any recent activity. If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contributions.