Question about how nightwatch tests are executed on Sauce Labs

[Silverfangy]

Hi guys,

I have a short question on how nightwatch tests will be executed by magellan on Sauce Labs. The background is that we have a certain number of VMs on Sauce Labs and in our magellan.json limited the number of workers to the exact same amount. So e.g. 10 VMs and max_workers 10. Still on Sauce Labs we get timeout errors when running this test suite (no other test suites were running in parallel).

Therefore I'd like to clarify: One nightwatch .test.js file corresponds to one worker. But if a test file includes several steps, every step takes one VM on Sauce Labs. So it can happen that even if you have 10 workers executing 10 test files, on Sauce Labs there will be more than 10 VMs used as the test files consist of more than one step. Am I correct with this? If not, could you maybe shortly explain how the different test steps in nightwatch play into this? Do you maybe have an idea how we could avoid running into these timeout issues? Is there a way to specify a timeout of more than 90 seconds (I think in the Sauce Labs documentation this is stated as the idle test timeout)?

Thanks in advance for your help :slightly_smiling_face: .

[Maciek416]

A quick rundown off the top of my head:

Workers/Steps:

If you specify a maximum number of workers as X, then Magellan will not use more than X VMs on your Saucelabs account. No matter what your individual test files in nightwatch do internally (regardless of steps), Magellan will consume them each individually with one worker at a time, with the total population of active workers at any given time never exceeding X. So if you have 100 test files, 10 workers, and 1 browser, each worker should on average (assuming no retries, etc) run about 10 tests. Hope that helps.

Timeout Issues:

This is a pretty complex issue that can depend heavily on many many factors and is the tip of the iceberg in test automation.

A primary timeout cause in our experience while developing Magellan in a secured corporate IT environment has been unreliable, poor, or blocked network access from the viewpoint of the Saucelabs VM. Make absolutely sure your Saucelabs VMs have reliable access to everything your application requests. When you review your Saucelabs result videos, try to figure out if the browser is still spinning trying to load the page. If it is, the Saucelabs VM may not have access to the resource at all and is simply blocking the loading of the page.

I'll also add that it's almost become a universally-held axiom over here that attempting to increase the Saucelabs idle test timeout is a kind of "test code smell". If your test has expensive I/O to do in preparation for the test, this I/O might be best to complete before Magellan runs (i.e. spinning up large-scale mocks, for example).

[archlichking]

No matter how many test steps there are in one test file, nightwatch only allocates one worker per file. Magellan plays in the same way. Magellan maps one worker per nightwatch file. So saucelabs VM and nightwatch test file is an one to one mapping.

[Silverfangy]

Thanks very much for your fast response! I got a little confused with the test files and steps as in Sauce Labs every test step is reported as a test run. But you are correct, they are never executed in parallel. So normally one test file should always account to one VM in use.

Thanks for clearing that up :slightly_smiling_face: .