Abort test on `waitUntil` failure when using it standalone with async/await.

[garg3133]

Description of the bug/issue

When we use the waitUntil command with command chaining (waitUntil command chained with some other command, not standalone), the test aborts correctly on waitUntil timeout. Using or not using async/await does not matter here. Ex.

browser
  .waitUntil(function() {
    browser.title();
    return false;
  })
  .pause();

But, if we use the waitUntil command without command chaining (standalone; waitUntil not chained with any other command) and with async/await, the test continues executing even after waitUntil timeout. Ex.

await browser.waitUntil(function() {
  browser.title();
  return false;
});
await browser.pause();

The test should abort in the second case as well.

This issue is created from https://github.com/nightwatchjs/nightwatch/pull/4123#discussion_r1524932212

Nightwatch.js Version

3.5.0

[garg3133]

This issue seems to be happening due to a side-effect of the queuing system.

In the case of chaining, if waitUntil timeouts, Nightwatch removes all the further commands from the queue and appends browser.end() command to the queue, which ends the test. This works well with the normal test runner but creates problem with Nightwatch Programmatic API:

Let's say, if we have the following in the test

await browser
  .waitUntil(function() {
    browser.title();
    return false;
  })
  .pause();

Now, if waitUntil command timeouts above, pause() will be removed from the queue and hence it will never get executed or even resolved. Due to this, the test execution will get stuck on this line forever.

While the above does not create any problem with Nightwatch or Mocha test runner (as they probably just dump the test case on failure <-- need to confirm this), it causes the test run to be stuck forever in case of Cucumber (which uses Nightwatch Programmatic API) or any other use of the Nightwatch Programmatic API.

---

But, in the case of direct use of async/await (without chaining), there's a weird side-effect of the above flow.

Let's say we have the following in the test:

await browser.waitUntil(function() {
  browser.title();
  return false;
});
await browser.pause();

Now, if waitUntil timeouts, the code below this line will run, where the waitUntil node will be resolved, the queue will be emptied and a new root node will be created.

But, as soon as the waitUntil node is resolved (when the JS event loop reaches the node resolution), the commands following the waitUntil command are put into the newly created queue before Nightwatch could put a browser.end() into the queue and due to this the test keeps running further without being aborted. And after seeing unfinished commands in the queue, Nightwatch does not put browser.end() into the queue till the very end.

[Temidayo32]

I couldn't reproduce this error on version 3.5.0. The waitUntil timeouts when used within an async/await

[garg3133]

@Temidayo32 Updated the issue description to be more clear. The test correctly aborts with async/await as well given that the waitUntil command is chained with some other command.

But, if we use async/await on the waitUntil command while not chaining it with some other command (as done in the second example in the issue description), the test does not abort on waitUntil failure and continues execution.

Nightwatch still displays the Timeout error in the second case, but the main thing is that Nightwatch should not continue with the test execution (it should not execute the .pause() command in this case).

[Temidayo32]

@garg3133 I think this issue might be a lot more bigger than just waitUntil. Here is a code beginning with navigateTo method first, and the test still didn't abort even though the navigateTo failed

async function myFunction () {
  await browser
      .navigateTo('https://ecosia.org/');
  await browser.waitUntil(async function() {
    const title = await browser.getTitle();
    return true;
  }, 2000);
  await browser.pause();
}

myFunction()

Here is what I observed about the queuing system in Nightwatch API. When a test is run initially, all the commands are read even before it start executing any of the commands. I am able to observe this by console logging the command name.

Running default: test/custom/testCustomAssertion.js

┌───────────────────────────── ✔ default: test/custom/testCustomAssertion.js ──────────────────────────────┐
│                                                                                                            │
│ navigateTo                                                                                                 │
│ waitUntil                                                                                                  │
│ pause                                                                                                      │
│ [Test/Custom/Test Custom Assertion] Test Suite                                                             │
│ ───────────────────────────────────────────────────────────────────────────────                          │
│ Error Error while running .navigateTo() protocol action: Cannot read properties of undefined               │
│ (reading 'navigate')                                                                                       │
│                                                                                                            │
│ ✖ TypeError                                                                                                │
│ Cannot read properties of undefined (reading 'navigate')                                                   │
│ Stack Trace:                                                                                               │
│   at MethodMappings.navigateTo (/home/danajayi/testnightwatch/node_modules/nightwatch/lib/transport/sele  │
│ nium-webdriver/method-mappings.js:186:29)                                                                  │
│   at /home/danajayi/testnightwatch/node_modules/nightwatch/lib/transport/selenium-webdriver/actions.js:5  │
│ 0:51                                                                                                       │
│   at Proxy.<anonymous> (/home/danajayi/testnightwatch/node_modules/nightwatch/lib/core/client.js:306:20)  │
│   at CommandInstance.command                                                                               │
│ (/home/danajayi/testnightwatch/node_modules/nightwatch/lib/api/protocol/navigateTo.js:53:34)             │
│   at /home/danajayi/testnightwatch/node_modules/nightwatch/lib/api/_loaders/command.js:182:29            │
│                                                                                                            │
│ getTitle                                                                                                   │
│ title                                                                                                      │
│                                                                                                            │
│ ✖ TypeError                                                                                                │
│ Cannot read properties of undefined (reading 'getTitle')                                                   │
│ Stack Trace:                                                                                               │
│   at MethodMappings.getPageTitle (/home/danajayi/testnightwatch/node_modules/nightwatch/lib/transport/se  │
│ lenium-webdriver/method-mappings.js:216:43)                                                                │
│   at /home/danajayi/testnightwatch/node_modules/nightwatch/lib/transport/selenium-webdriver/actions.js:5  │
│ 0:51                                                                                                       │
│   at Proxy.<anonymous> (/home/danajayi/testnightwatch/node_modules/nightwatch/lib/core/client.js:306:20)  │
│   at CommandInstance.command                                                                               │
│ (/home/danajayi/testnightwatch/node_modules/nightwatch/lib/api/protocol/title.js:19:34)                  │
│   at /home/danajayi/testnightwatch/node_modules/nightwatch/lib/api/_loaders/command.js:182:29            │
│                                                                                                            │
│ Paused...                                                                                                  │
│ Press <space> or F10 to step over to the next test command and pause again.                              │
│ Press Ctrl+C to exit.                                                                                      │
│ Press any other key to RESUME.                                                                             │
│                                                                                                            │
│ Using: chrome (123.0.6312.86) on LINUX.                                                                    │
│                                                                                                            │
│ end                                                                                                        │
│ session                                                                                                    │
│                                                                                                            │
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘

As can be seen, when the test first start running, all the commands (navigate, waitUntil and pause) are added to the command queue. At this point, none of the commands, I think, isn't executed yet. Thereafter this, is when the executing comes. And as can be seen, it when it start running each command, it calls the commands called by the initial ones. For instance, it calls getTitle and and then title when executing waituntil.

Also, as can be seen, when the navigateTo fails, but the test doesn't abort, and proceeds to execute waitUntil which also fails too. The reason I think the test don't abort when a command fails, is probably because all the commands have been added to the command queue from the beginning.

[AbhisekOmkar]

browser .waitUntil(function() { browser.title(); return false; }) .pause();

what is the path of this code

[capGoblin]

Here is what I observed about the queuing system in Nightwatch API. When a test is run initially, all the commands are read even before it start executing any of the commands. I am able to observe this by console logging the command name.

  it('Demo test ecosia.org', async function (browser) {
    await browser
      .navigateTo('https://ecosia.org/');
    await browser.waitUntil(function () {
      browser.title();
      return false;
    }, 2000)
    console.info("Tests")
    await browser.pause()
});

@Temidayo32 if you try running this test after setting silent: false in nightwatch config, you will notice each cmd being run when logging the cmd names. The cmds are added by this function and it is traversed & run by the same function right here

Now, if waitUntil timeouts, the code below this line will run, where the waitUntil node will be resolved, the queue will be emptied and a new root node will be created.

And you can see that this test continues executing even after waitUntil timeout as mentioned by @garg3133