Feature: Top-level suite configuration

[devpaul]

Currently Intern's configuration recognizes two modes of testing. These modes break down into Selenium/WD tests (functionalSuites) and tests that can run in a client or browser platform (suites).

{
// Non-functional test suite(s) to run in each browser
suites: [ 'tests/unit/all' ],

// Functional test suite(s) to run in each browser once non-functional tests are completed
functionalSuites: [ 'tests/e2e/all', 'tests/functional/all' ]
}
``` This was a good way for defining the standard 2-stage paradigm of unit and functional testing. However, as we add more types and stages of testing this style of configuration starts to become cumbersome. For instance if I wanted to define a configuration with unit tests, functional (component) tests, end-to-end tests, visual regression tests, scripted QA tests, and integration/system tests the `suites` and `functionalSuites` configuration no longer support identifying each of these test runs.

In order to support more stages of testing, we need to decouple the mode of testing (runner/client) from the stage or type of test run (unit, functional, e2e, etc...). A possible configuration could look like this:

{ stages: { unit: { suites: [ 'tests/unit/all' ], platform: 'client' }, // functional tests of individual components functional: { suites: [ 'tests/functional/all' ], platform: 'webdriver' } // blackbox feature testing (requires VPN & server) e2e: { suites: [ 'tests/e2e/all' ], platform: 'webdriver' }, // visual regression tests (requires VPN & server) visual: { suites: [ 'tests/visual/all' ], platform: 'webdriver' }, // tests generated by QA using Intern recorder (requires VPN & server) acceptance: { suites: [ 'tests/qa/all' ], platform: 'webdriver' } default: { suites: [ 'tests/unit/all', 'tests/functional/all' ] platform: 'webdriver' } } } `` From the command line we can run tests usingintern unit --config=tests/internorintern --config=tests/internto run the default set of tests. Being able to use custom configurations for each stage of testing may also help test other platforms, such as Electron or Appium. It also has the benefit of replacingintern-runnerandintern-client` with a single configuration-driven solution.

I'd love to hear feedback on this before starting on a PR. Thanks.

[jason0x43]

intern-runner and intern-client are separate because they work differently, not because they inherently serve different styles of tests. Either of them may be used to run unit tests, or e2e tests, or visual regression tests; the proper engine really depends on what work needs to be done (e.g., if a visual regression test calls an external service, it will run fine using the Node client, but if it needs to inject custom code into a page, it may need the WebDriver runner) .

The same thing goes for suites vs functionalSuites -- the distinction is mechanical (WebDriver vs non-WebDriver), not necessarily based on the type of test. It's quite possible a user may have unit tests that need to run in a browser (so intern-client wouldn't make sense), or they may have some e2e tests that run fine completely in the Node client, and some that need a browser.

To to this, we would need to provide a way to annotate tests so that Intern will know what type of environment is required (or desired, in case they're different). There's definitely some worthwhile possibility here, although it would require some significant internal changes. It would also probably work better if Intern could pre-enumerate its tests so that it could select the necessary engine(s) up front; that's planned for some indeterminate future release. A short term possibility is just to use separate configs for each testing stage (where the user could handle which tests were run, and how they were run, in the stage config).

[sholladay]

the distinction is mechanical (WebDriver vs non-WebDriver)

I think what would make this more intuitive for people is if it were intern --browser instead of intern --webdriver and say that functionalSuites requires that. Then it is more about the what and less about the how.

[devpaul]

Sure, intern-runner and intern-client have different capabilities. I'm looking for a way to call out to these different mechanics for each type of testing strategy needed in a project using the 'platform' property.

In the above configuration a test stage with platform: client would use the current intern-client mechanism and platform: webdriver would use the intern-runner mechanism. Maybe the property names should be tweaked (it's only a first pass :), but essentially I'd like to change how we instrument our runner so it is driven by configuration rather than the execution of one of intern-runner or intern-client.

It would give us several benefits:

1. support n-stages of test with a single Intern configuration
2. reduce confusion regarding which executable should be run by pulling that choice into the configuration
3. ability to list the configured stages of test via the cli
4. explicitly configure a stage of test with instrumentation
5. simplify url and command line arguments
6. later, add other test instrumentation platforms in the config (e.g. Cucumber or WD.js)

We currently control the mechanics of how we test using intern-runner and intern-client. Long term I envision us using a single cli with instrumentation happening through configuration rather than execution.

The process would look something like

1. cli starts Intern
2. reads configuration
3. selects instrumentation (runner/client)
4. runs tests using the instrumentation

This type of selective instrumentation will also likely set us up for a plugable instrumentation architecture by allowing the configuration to define additional 3rd party instrumentation plugins.

@jason0x43 regarding annotating test, I'm not sure I follow. Is this similar to the platform property in identifying whether the client or runner should be used? Could you explain more what you were thinking here?

[jason0x43]

The term "instrumentation" is already used for code coverage; "test runner" or "engine" would be preferable when talking about the thing that's running the tests. Unfortunately we kind of already used "runner" for "webdriver runner" (vs. "Node client"). So...engine? Test runner?

This seems like more of a usage pattern than a feature. I mean, most of this can already be handled easily via external scripts, or in the Intern config itself. Also, the concept of "stages" will be unique to the user, and it may not be something with universal appeal.

It's pretty trivial to select suite sets in a single config:

define([ 'intern' ], function (intern) {
    var stage = intern.args.stage;
    var config =  {
        // stuff
        suites: [],
        functionalSuites: []
    };

    switch (stage) {
    case 'visual':
        config.suites = [ /* suites */ ];
        config.functionalSuites = [ /* other suites */ ];
        break;
    case 'e2e':
        // ...
        break;
    }
});

Getting some sort of automatic or configured test runner selection is a bit trickier. There isn't a good way for Intern to know whether unit tests need a browser or can run in the Node client (and the user might prefer one in any case). There are also cases where you may want to do both (like Intern's self tests).

[sholladay]

There isn't a good way for Intern to know whether unit tests need a browser or can run in the Node client (and the user might prefer one in any case). There are also cases where you may want to do both (like Intern's self tests).

How about we add one more array to the config? The problem lies in the fact that suites is ambiguous, unlike functionalSuites. But that is easy to solve. We split it up into browserSuites and nodeSuites.

[jason0x43]

That's a possibility. We could have something like nodeSuites, browserSuites, and webdriverSuites. It could also simplify how we handle the distinction currently, which usually involves a lot of intern/dojo/has!host-node?... MIDs.

[devpaul]

It sounds like we're going to hold on this until around/after Intern 4 is released. I am still in favor of engine and suite selection via configuration rather than programmatically defining what engines and suites should run. All of these are problems that were mentioned here, engine selection (runner vs client), multiple suite configuration, and test selection (i.e. has!host-*), are very common for our users that I would like to resolve.

We could have something like nodeSuites, browserSuites, and webdriverSuites

While selecting tests that run in node, browser, or webdriver are common filters, we should aim to generalize test filtering using dojo-has so engineers can say things like 'only run this suite of tests if you're in a browser with webvr support'. This would preferably be configuration driven.

Finally, just to play devil's advocate for following the programmatic path...

Intern provides information about what engine it is running so technically the separate suites and functionalSuite could be unified into a single list of suites and defined using the above switch statement example or users could write an AMD plugin or use dojo/has to select tests to run based on the engine detected. I only mention it because I think it highlights the odd state I feel we are in between how we've determined what should be configuration driven and what should be programmatic.

[jason0x43]

The main issue I have is one of simplicity vs complexity. Currently, the available tools (Intern features) are pretty straightforward. We have a Node client and browser client and a WebDriver runner. You select your environment, and Intern runs the relevant tests. (There is some conflation between node vs browser suites because of the single suites list, but this could be alleviated by breaking it into separate lists.) If you need to restrict which tests run in which environment, this is very easy to do at either the suite or test level using whatever means the test writer desires (such as dojo-has flags), and it's easy to look at a list of suites or tests and see when they will run.

A specialized lifecycle-based configuration layer adds complexity to Intern without actually achieving the goal of relieving the test writer of having to be aware of test runners since there is no direct mapping between lifecycle stage and test runner. That not to say that this sort of configuration wouldn't be valuable in some scenarios, but it may fit better in a front end rather than Intern's core.

[jason0x43]

I don't mean to sound like I think Intern doesn't need improvement. It could certainly be made more approachable and easier to configure. I just think we may be better served by focusing on making the core functionality easier to understand and utilize than we would by adding another layer of configuration.

[devpaul]

We're not adding an additional layer of configuration so much as moving choices that already need to be made by the developer into the configuration. Instead of making engine selection an external process that happens on the command line we are better served moving this into the configuration.

Today if an engineer in charge of testing wants to have multiple test lifecycles that use various engines they need to write an Intern configuration file for each one and inform other engineers on their team which configuration to use and which engine to select. There are external solutions to this issue of information sharing (i.e. use npm scripts; grunt; gulp; intern cli; command line). These various solutions do nothing to reduce complexity on the whole; they only help compartmentalize it.

Instead we should move these choices into a configuration file that can be used as a single source of information.

stages: {
        unit: {
            suites: [ 'tests/unit/all' ],
            engine: 'client'
        },
        // functional tests of individual components
        functional: {
            suites: [ 'tests/functional/all' ],
            engine: 'webdriver'
        },
        default: {
                suites: [ 'tests/unit/all', 'tests/functional/all' ]
                engine: 'webdriver'
        }
}

The above configuration moves engine and suite selection into a single configuration file, which could in turn be queried by the cli; e.g. intern list would return unit, functional, default which would be run using intern unit or intern functional or just intern to run the default stage. This may add some complexity to the configuration, but it removes the need for all of the engineers on a team to have a deep understanding of Intern's engines. They no longer need to look at Intern's configuration files to understand what suites are being run and they don't need to know when suites and functionalSuites are ran in conjunction to a particular engine.

Intern is a very capable and complete testing framework. If we are going to have complexity, my preference is to keep it in a single location and use the cli to simplify usage for the overall team. We are not well served by keeping Intern simple if it makes it harder to use for everyone else.

[jason0x43]

tl;dr This proposal would fit better in intern-cli.

This may add some complexity to the configuration, but it removes the need for all of the engineers on a team to have a deep understanding of Intern's engines. They no longer need to look at Intern's configuration files to understand what suites are being run and they don't need to know when suites and functionalSuites are ran in conjunction to a particular engine.

In the projects I've seen and worked on, that kind of knowledge is abstracted away, as much as makes sense, with grunt, scripts, maven, or whatever test driver the project uses. Yes, the test writers need to have some understanding of how Intern works, but the proposed stage config does really help with that.

Today if an engineer in charge of testing wants to have multiple test lifecycles that use various engines they need to write an Intern configuration file for each one and inform other engineers on their team which configuration to use and which engine to select. There are external solutions to this issue of information sharing (i.e. use npm scripts; grunt; gulp; intern cli; command line). These various solutions do nothing to reduce complexity on the whole; they only help compartmentalize it.

I think this is the key point where I disagree with this proposal. I feel like this type of opinionated configuration should be handled with external solutions (including intern-cli). Everyone has a favorite interface. Some people are fine with the base Intern scripts, some prefer grunt (and we have a grunt task), and we've gotten requests a library version of Intern to allow for even more flexibility. External tools provide myriad ways of configuring the testing process, and that's great. Core Intern should focus on providing a simple, clear, and reasonably unambiguous API for running and managing tests. The CLI that's built into Intern is fairly raw, and that's fine; it should provide access to Intern's features, but it probably shouldn't provide a lot of functionality on top of that.

[devpaul]

I think this is the key point where I disagree with this proposal. I feel like this type of opinionated configuration should be handled with external solutions

we've gotten requests a library version of Intern to allow for even more flexibility

Core Intern should focus on providing a simple, clear, and reasonably unambiguous API for running and managing tests.

I think we should discuss how we envision fixing usability issues with Intern. From what you said above it sounds like if we had to do it all over again that Intern would only have a programmatic API and we would produce external tools like intern-cli to handle configuration and execution.

We should use our review time on the roadmap to discuss where Intern 4 and 5 should take us. If we are to reseat this feature request in intern-cli it wouldn't change the feature or the changes we'd need to make to Intern to support it. And if we plan on offering an programmatic API we'll still need to do the same work we discussed above to move engine selection internal to Intern as part of the APIs.

And if in some parallel universe Intern did only have a programmatic API and relied on external tools like intern-cli and (a separate) grunt-intern and (another) intern-browser-runner (browser tests) for executing that API those tools would still be relatively tightly coupled by their configuration since their use would need to be interchangeable. So if we were to move this feature to intern-cli and add a super-set of configuration, I would assume we would have to add it to Intern's grunt task as well as the browser runner.

Long story short, we need to revisit our vision for Intern before we come back around to this. I am OK moving this feature as long as it produces a cohesive ecosystem, but I don't think it is possible to move this feature out of Intern's core without also externalizing Intern's configuration.

[jason0x43]

I think we should discuss how we envision fixing usability issues with Intern. From what you said above it sounds like if we had to do it all over again that Intern would only have a programmatic API and we would produce external tools like intern-cli to handle configuration and execution.

This is a natural direction for Intern to take; there's no need to "do it all over again". It lets core Intern focus on providing testing facilities without shackling it to any particular UI or testing methodology.

If we are to reseat this feature request in intern-cli it wouldn't change the feature or the changes we'd need to make to Intern to support it.

This feature could be implemented in a UI without requiring any changes to Intern.

And if we plan on offering an programmatic API we'll still need to do the same work we discussed above to move engine selection internal to Intern as part of the APIs.

There is no requirement that Intern handle engine selection. It certainly could (to some extent), but the UI could just as well. The main goal of creating a feature like this should be to improve usability for end users. Assuming automatic engine selection is something that's a significant pain point, it's not going to matter to an end user where that's handled in Intern's stack, just that it's handled. From an architectural perspective, I'm not sure that it's better to have Intern perform automatic engine selection vs letting the UI (or the user) make that decision.

And if in some parallel universe Intern did only have a programmatic API and relied on external tools like intern-cli and (a separate) grunt-intern and (another) intern-browser-runner (browser tests) for executing that API those tools would still be relatively tightly coupled by their configuration since their use would need to be interchangeable.

Intern's CLI is already a pretty thin layer over the underlying code; exposing an API won't be all that difficult. In any case, many users already rely on something else to provide the user-facing UI, like grunt, npm scripts, maven, etc., so this isn't a huge paradigm shift.

Long story short, we need to revisit our vision for Intern before we come back around to this. I am OK moving this feature as long as it produces a cohesive ecosystem, but I don't think it is possible to move this feature out of Intern's core without also externalizing Intern's configuration.

One of the main benefits of separating the UI from the core is that experimentation with new UI concepts becomes both easier and less risky. No changes to Intern's core (that I can see) are required to implement this feature, so there's no reason it couldn't be implemented in a standalone UI right now (or added to intern-cli).

[devpaul]

One of the main benefits of separating the UI from the core is that experimentation with new UI concepts becomes both easier and less risky

This ticket started with adding to Intern's configuration to address a common use case and well-defined scenario and the level of effort has increased to creating an external UI. Creating an external UI with the same features of Intern means adding the configuration then building a cli, a grunt task, and a browser runner that can transform the new configuration into one Intern understands and then maintaining it separately as Intern is updated. This is an unrealistically high bar to set for any feature.

[jason0x43]

During some offline discussions, the following modifications to Intern were proposed that would allow the proposed feature to be implemented:

• Split the suites list into browserSuites and nodeSuites. This will allow a front end to identify which engine(s) would be required to execute the suites in a config.
• Give intern-cli the ability to list available config files. This means that the config files would need to live in a standard location.
• Add support for a local config file for intern-cli, something like an .internrc. This will allow for project-specific modifications to intern-cli's behavior, such as changing where the app looks for the default config file.
• Add a standard metadata property to the config files. This will allow interested parties (like intern-cli) to display useful information about the available configs.
• Give the browser client a query mode with the ability to list available configs (those with browser suites, at least). This will require some additional capabilities in the test proxy server.

With these updates, high-level organization can be implemented using individual config files. These can be queried using standard OS facilities such as ls, as well as intern-specific features. Each config config provide metadata describing the purpose and capabilities of the config, and will contain any environment-specific setup related to the config.

[jason0x43]

Closing this as its specific to Intern 3, and because Intern 4 essentially supports the original proposal with child configs.