Handling test dependencies (tests that depend on other tests)

[simonvizzini]

I'm just getting started with Nightwatch and End-to-End testing in general, so it's highly likely that I'm missing something.

I'm trying to figure out how to structure my tests into groups and tags. In the end I'd like to have a couple of test suites I could run individually, something like: a "full" test suite that runs every test, a "quick" test suite that just runs some basic tests (these tests may be split up between different groups), and maybe other, feature specific test suites.

I fear that I will sooner or later run into limitations with just grouping/tagging my tests, because I'll most likely have tests that will depend on other tests, so the order of execution would matter.

I think what I would need is a way to define test suites as an array of test files, which would then get executed in the specified order, instead of just specifying src_folders, groups or tags. Alternatively I could number my groups/testsfiles to ensure order of execution, but that doesn't feel like a good solution.

I did a quick google search and it seems that I'm the only one that would like to have some kind of dependencies between tests. I probably have a fundamental misunderstanding of how end-to-end tests should be implemented. So please let me know if what I'm trying to do is just wrong :smiley:

[garethdn]

+1 I have the same problem and think I might be missing some kind of fundamental principal.

My scenario is that I need to access a list of users but I need to be authenticated before I can do this. The test for accessing users is in my user.js spec and the tests for logging in is in my login.js spec.

Basically, before every one of my tests I need to be logged into the system.

[simonvizzini]

@garethdn: Logging in before every test should be fairly easy, you could put some logic that checks if you are logged in or not in the before or beforeEach functions. If you detect that you are not logged in you could then run your login code, wait until you are logged in and give back control back to nightwatch to execute your tests (check async beforeEach and beforeAfter in the documentation for this). So this one was kind of easy to handle. But I'm running into other dependency problems, for example a adminPanel.js spec includes a test that creates a new user. But now other specs may depend on this newly created user to be present in the system (outside of the adminPanel spec).

In the meantime I have moved on to protractor (it's designed for testing Angular sites, but works for non-Angular sites as well by disabling the angular sync feature). It allows me to define multiple test suites through the config file, each suite containing an array of folders and/or files (with glob support) with specs to run, giving quite some flexibility and control in setting up multiple test suites, where I can put depending specs in the right order.

Hope this helps somewhat.

[garethdn]

@simonvizzini Thanks for that, I've implemented login in the before methods for my tests but am running into the same problem as you describe in your OP. I have an application tag that contains two specs at the moment, createApplication and editApplication specs. Obviously the editApplication spec depends on the createApplication spec having been run beforehand. This will probably work fine, purely by coincidence, because the specs seems to execute alphabetically by file name - but it's not way a sustainable way to define the execution order.

[beatfactor]

Usually dependencies can be handled by the continuos integration system (such as teamcity or jenkins). You can split the tests by group or by tag and create different build configurations and then specify the dependencies.

On Wednesday, April 1, 2015, garethdn notifications@github.com wrote:

@simonvizzini https://github.com/simonvizzini Thanks for that, I've implemented login in the before methods for my tests but am running into the same problem as you describe in your OP. I have an application tag that contains two specs at the moment, createApplication and editApplication specs. Obviously the editApplication spec depends on the createApplication spec having been run beforehand. This will probably work fine, purely by coincidence, because the specs seems to execute alphabetically by file name - but it's not way a sustainable way to define the execution order.

— Reply to this email directly or view it on GitHub https://github.com/beatfactor/nightwatch/issues/391#issuecomment-88548137 .

[garethdn]

@beatfactor I understand that tests can be split by tag or group but let's say I have a user tag which contains specs for create, edit and finally delete. How do I ensure that the specs are executed in this order without actually naming the files something like, 01createUser.js, 02editUser.js and 03deleteUser.js?

EDIT: As an example, I'm fairly sure that this is why Slide 8 at http://www.slideshare.net/bastiao/nightwatcherjs is using that specific naming convention.

[beatfactor]

I don't have a solution for this problem. I personally think that test suites shouldn't depend on others, but I am willing to consider a concrete suggestion for an enhancement. So if anybody has one please create a new issue.