Filter or grep specific tests to run (like mocha grep pattern)

[hannah06]

Current behavior:

Run specific files but cannot choose specific single/multi tests.

Desired behavior:

Choose expected tests to run, like mocha grep pattern. Though we put a group of tests in a spec file, but sometimes we just want to run parts of them, so we need filter them out.

[jennifer-shehane]

From mocha docs:

-g, --grep <pattern> only run tests matching <pattern>

The --grep option when specified will trigger mocha to only run tests matching the given pattern which is internally compiled to a RegExp.

Suppose, for example, you have “api” related tests, as well as “app” related tests, as shown in the following snippet; One could use --grep api or --grep app to run one or the other. The same goes for any other part of a suite or test-case title, --grep users would be valid as well, or even --grep GET.

described ('api', function() {
  describe('GET /api/users', function() {
    it('respond with an array of users', function() {
      // ...
    });
  });
});

describe('app', function() {
  describe('GET /users', function() {
    it('respond with an array of users', function() {
      // ...
    });
  });
});

[smccarthy]

+1

[kuceb]

I personally like the idea of tagging tests, because you don't have to worry about organization as much. It would be nice to find all the tests related to something using only the concept of Tagging, as explained in the mocha wiki below:

https://github.com/mochajs/mocha/wiki/Tagging

Tagging

visionmedia edited this page on Oct 10, 2012 · 2 revisions Mocha's --grep feature may be used both on the client (via ?grep=) and server-side. Recent releases of Mocha allow you to also click on the suite or test-case names in the browser to automatically grep them. The concept of Tagging utilizes regular grepping, however may be a useful way to keep related tests in the same spot, while still conditionally executing them.

A good example of this is if you wanted to run slow tests only before releasing, or periodically. You could use any sequence of characters you like, perhaps #slow, @slow to tag them as shown here:

describe('app', function(){
  describe('GET /login', function(){
    it('should respond with the login form @fast', function(){

    })
  })

  describe('GET /download/:file', function(){
    it('should respond with the file @slow', function(){

    })
  })
})

To execute fast tests only then you may do --grep @fast. Another alternative is to only tag @slow, and utilize --grep @slow --invert to invert the grep expression.

[MaaikeFox]

+1! We would really be helped with this feature. We don't use cucumber so we can not use the tag feature for that. And we want to stay close to the original Cypress without hacking things in ourselves.

We have a complex system with several services and applications working together. Cypress covers the End-to-end tests on all of this together. So we want to be able to select groups of tests on CI (Gitlab) to not have to run all tests every time but only a relevant selection of the tests.

[fakiolinho]

This would be a super useful feature especially when we are testing a big app so we want to split our e2e tests somehow into suites after tagging them accordingly.

[jesusochoahelp]

This would be really useful. Being able to only run smoke test TCs from different files makes organization easier.

[rachelsevier]

This would be so useful!

[ziyailkemerogul]

I've been searching for this nice feature in Cypress and found out this post, thanks to @hannah06 .

Here are my thoughts on this feature:

I have lots of test cases with different priorities and I want to tag them according to their priorities. Actually, it's not for just a priority like; "Critical", "High", "Medium", "Low", I can use tags like "Functional", "Non-Functional", "Smoke", "UI", etc. So, I can group and run my tests, how I want or how I structure. Sometimes I just want to be able to run only the "Critical" cases, or "Critical" and "UI" cases. By this way, we will be able to structure more dynamic tests.

[xeger]

I'm using cypress-cucumber-preprocessor which translates Gherkin to JS and supports Cucumber tag expressions. It relies on the TAGS env variable to provide a tag filter.

For those using pure JS, you could emulate this behavior by wrapping the Mocha it method:

// check Cypress.env('TAGS') which might be e.g. "@foo and not @baz"
withTags('@foo @bar').it('works well', () => { cy... } )

The cucumber-js tag helper API does all of the hard parsing/matching and could be plugged into this solution.

If you wanted something more magical feeling, you could write a preprocessor plugin that scans the file for magic comment lines or special describe blocks or something. (Probably overkill!)

[jweingarten]

Here is my workaround:

Test example:

import TestFilter from '../../support/util/test-filter';

describe('Dashboard', function() {
  TestFilter.any(['smoke', 'pr'], () => {
    it('loads and validates dashboard page', function() {
      cy.dspVisit('/app/dashboard/list');
      cy.PageObjects.DashboardPage().validate();
    });
  });
});

TestFilter class:

const TestFilter = {
  any(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');
    const found = tags.some((r) => selectedTags.indexOf(r) >= 0);

    if (found) {
      testFn();
    }
  },

  notIn(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');
    const found = tags.some((r) => selectedTags.indexOf(r) >= 0);

    if (!found) {
      testFn();
    }
  },

  all(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');

    if (
      tags
        .map((tag) => {
          return tags.length === selectedTags.length && selectedTags.indexOf(tag) !== -1;
        })
        .reduce((acc, cur) => acc && cur, true)
    ) {
      testFn();
    }
  }
};

export default TestFilter;

And you run your tests by passing in TEST_TAGS.

[wjcheng1]

any updates on this? would love to have this. thanks!

[StormPooper]

I ended up with this after experimenting with it today, coincidentally. It will look through the entire suite and skip anything that doesn't match your tags, including the parents if there's no tests that need to run - plus it doesn't need anything importing in individual tests, which is nice too. I've simplified it slightly from our production code, so let me know if there are any mistakes.

Install the cucumber-tag-expressions node module and add or import this in your /support/index.js:

import { TagExpressionParser } from 'cucumber-tag-expressions';
const tagParser = new TagExpressionParser();

before(function() {
    this.test.parent.suites.forEach(checkSuite);
});

const shouldSkip = test => {
    const tags = Cypress.env('tags');
        if(!tags) return;
    const tagger = tagParser.parse(tags);
    return !tagger.evaluate(test.fullTitle());
};

const checkSuite = suite => {
    if (suite.pending) return;
    if (shouldSkip(suite)) {
        suite.pending = true;
        return;
    }
    (suite.tests || []).forEach(test => {
        if (shouldSkip(test)) test.pending = true;
    });
    (suite.suites || []).forEach(checkSuite);
};

Then you can use it in your tests by setting a tags environment variable in Cypress (I do this with npm-run-all and it's argument placeholders within our script definitions, but you can use any method you like to set the tags). The variable should be a string using the Cucumber tag expressions syntax format, so "@foo and @bar" for example.

You use the tags in your test names, like so:

describe('set of tests @foo', () => {
    it('test @bar', () => {...});
    it('another test @baz', () => {...});
    describe('sub-suite @xyzzy', () => {...});
});

[bahmutov]

if you use JavaScript specs, take a look at the plugin https://github.com/bahmutov/cypress-select-tests that allows one to select tests to run by filename or by substring

 ## run tests with "works" in their full titles
 $ npx cypress open --env grep=works
 ## runs only specs with "foo" in their filename
 $ npx cypress run --env fgrep=foo
 ## runs only tests with "works" from specs with "foo"
 $ npx cypress run --env fgrep=foo,grep=works
 ## runs tests with "feature A" in the title
 $ npx cypress run --env grep='feature A'

[x-yuri]

@jennifer-shehane or anybody else, how do I make use of the --grep switch?

$ ./node_modules/.bin/cypress run -g 'some pattern'                                                                             

  error: unknown option: -g
...

[jennifer-shehane]

@x-yuri It is not supported. This issue is requesting this to be added as a feature. You could use this plugin today as a workaround: https://github.com/bahmutov/cypress-select-tests

[deshdeep-airwallex]

Here is my workaround:

Test example:

import TestFilter from '../../support/util/test-filter';

describe('Dashboard', function() {
  TestFilter.any(['smoke', 'pr'], () => {
    it('loads and validates dashboard page', function() {
      cy.dspVisit('/app/dashboard/list');
      cy.PageObjects.DashboardPage().validate();
    });
  });
});

TestFilter class:

const TestFilter = {
  any(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');
    const found = tags.some((r) => selectedTags.indexOf(r) >= 0);

    if (found) {
      testFn();
    }
  },

  notIn(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');
    const found = tags.some((r) => selectedTags.indexOf(r) >= 0);

    if (!found) {
      testFn();
    }
  },

  all(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');

    if (
      tags
        .map((tag) => {
          return tags.length === selectedTags.length && selectedTags.indexOf(tag) !== -1;
        })
        .reduce((acc, cur) => acc && cur, true)
    ) {
      testFn();
    }
  }
};

export default TestFilter;

And you run your tests by passing in TEST_TAGS.

This looks good and works. How I can write a CLI script for it to filter in Jenkins job "cypress-webpr": "cypress run --record --key $CYPRESS_KEY --parallel --spec \"cypress/integration/**/!(*.wip).spec.js\" --env TEST_TAGS=pr -b chrome" Does this looks right? it seems to ignore the --env variable TEST_TAGS in CI

[smart625]

@jennifer-shehane any updates on this? It's very helpful for our project. Thanks!

[tomardern]

I've managed to do this in a simpler way:

  on('file:preprocessor', file => {
    if (config.env.specname && file.filePath.indexOf(config.integrationFolder) > -1) {
      const contents = fs.readFileSync(file.filePath, 'utf8');
      const modified = contents.replace(
        /(^\s.?)(it)\((.*$)/gim,
        (a, b, c, d) => `${b}it${RegExp(config.env.specname, 'gmi').test(d) ? '' : '.skip'}(${d}`
      );
      fs.writeFileSync(file.outputPath, modified, 'utf8');
    }
    return Promise.resolve(file.outputPath);
  });

To use: cypress run --env specname='NAME HERE'

This will add '.skip' to any tests which do not include the specname defined in the environment variable

[x-yuri]

If you want to temporarily focus on a couple of tests:

describe('...', function() {
    specify('test1', () => {
        console.log('test1');
    });
    specify('test2', () => {
        console.log('test2');
    });
    const t3 = specify('test3', () => {
        console.log('test3');
    });
    console.log(t3.fullTitle());
    mocha.grep(/test[12]/);
});

[lulubobst]

Please add tagging

[Rulexec]

My current solution without wrapping test code at all:

// support/index.js

let testFilter;

global._testFilter = function(x) {
    if (typeof x === 'function') {
        testFilter = x;
    } else {
        // `x` must be RegExp
        testFilter = function({ fullTitle }) {
            return x.test(fullTitle);
        };
    }
};

// Replace `context`/`it` to collect full title and filter them

let suiteNames = [];

let oldContext = global.context;
global.context = function(name) {
    suiteNames.push(name);

    let result = oldContext.apply(this, arguments);

    suiteNames.pop();

    return result;
};

let oldIt = global.it;
global.it = function(name) {
    if (!testFilter) return oldIt.apply(this, arguments);

    let fullTitle = suiteNames.join(' ') + ' ' + name;

    if (!testFilter({ fullTitle })) return;

    return oldIt.apply(this, arguments);
};

Then in the beginning of the test file simply add something like _testFilter(/user should be/);. It is possible to read regexp from environment variable in the support file, if you want.

But there anyway will be empty contexts.

[bahmutov]

Proposal

When Cypress runner finishes collecting tests its creates a single object and then starts running the tests. We can insert an async operation into this gap. We can pass the tree of collected tests to the user's async function in the plugins file. The user can filter the tests by name in any way desired: using CLI arguments or by looking up which tests to run via API requests, or by reading the names of the tests from a file.

// plugins file
// run all tests
on('filter:tests', (rootSuite) => {
  // rootSuite
  //   tests: [test objects]
  //   suites: [recursive suites]
  // each test object has "title"
  return Promise.resolve(rootSuite)
})

or run just the first test from the root suite

on('filter:tests', (rootSuite) => {
  rootSuite.suites.length = 0
  rootSuite.tests.length = 1
  return Promise.resolve(rootSuite)
})

The runner code on receiving an object from 'filter:tests' callback with filtered tests will go through its normalized tree of tests and remove any tests that are NOT in the returned tree of tests.

Inspiration: run-time filtering of Mocha's tests https://glebbahmutov.com/blog/filter-mocha-tests/

[jimpriest]

Would like to see this as well. I'm converting some of my Robot Framework tests over and am missing this feature. I don't think it should be tied to a file name as I want to be able to change 'tags' without renaming things.

[goodliff]

+1 for tags, also the ability to only run the tests/files returned from the search in the GUI rather than the "Run all specs" button.

[xeger]

FWIW, here is a toy implementation of tag-based selective test execution; as you can see, it's fairly easy to do with no help from Cypress itself: https://codesandbox.io/s/immutable-monad-gz839

[thviQit]

Sorry for the spam, I just thought I would share my workaround. I created a file in Cypress's support folder called filterTestsByTags.ts. Here I implemented an IIFE that will overwrite the it function if tags are provided in Cypress.env("Tags"). In the test result, the skipped tests will be listed as Pending. In this way, you can tag your tests as with mocha's grep, and when Cypress get their own implementation done, that works with grep, it should be easy to upgrade.

Disclaimer Not tested with Cypress 4

support/filterTestsByTags.ts:

import { TestFunction } from "mocha";

(function () {
    if (!Cypress.env("Tags")) {
        return;
    }

    const tags = Cypress.env("Tags").split(",");
    let orgIt = it;
    let filterFunction = function (title, fn) {
        if (tags.find(t => title.indexOf(t) === -1)) {
            fn = null;
        }
        orgIt(title, fn);
    }
    let filteredIt = filterFunction as TestFunction;
    filteredIt.skip = orgIt.skip;
    filteredIt.only = orgIt.only;
    filteredIt.retries = orgIt.retries;
    it = filteredIt;
})()

support/index.ts:

import './filterTestsByTags'

CLI: npx cypress run --env Tags=#slow,#critical

[indrajitbnikam]

@thviQit How are you using tags in tests? At what level are you defining tags.

I am using @cypress/webpack-preprocessor to write my tests in .ts file just like yours 😊

[thviQit]

@indrajitbnikam

describe("My component", function(){
    // tagged with #slow
    it("loads an entire database #slow", function()
        // cypress test here
    })
})

Then in our build pipeline I can schedule test runs for the slow tests. npx cypress run --env Tags=#slow

You could also update filteredTestsByTags to handle excludes so you don't have to tag all the other tests as well. My filter function is also pretty simple, and could be updated to use regex like grep.

[coder-shanks]

@thviQit this does not work with cypress 4.1, I am using typescript for writing tests

[coder-shanks]

@jennifer-shehane @bahmutov When can we expect this as a official feature of Cypress?

[thviQit]

@coder-shanks Well I found out Cypress has a bug, so it cannot parse --env if it's a comma separated list, so now we do this instead:

npx cypress run --env tags='#tagone #tagtwo'

or

npx cypress run --env exclTags='#tagone #tagtwo'

Right now my filter looks like this in TypeScript:

import { TestFunction } from "mocha";

(function () {
    if (!Cypress.env('tags') && !Cypress.env('exclTags')) {
        return;
    }

    const envTags = Cypress.env('tags') ?? '';
    const envExclTags = Cypress.env('exclTags') ?? '';
    const hasTags = envTags !== '';
    const hasExclTags = envExclTags !== '';

    // Don't filter if both is defined. We do not know what is right
    if (hasTags && hasExclTags) {
        console.log('Both tags and excluding tags has been defined. Not filtering');
        return;
    }

    const tags = hasTags ? envTags.split(' ') : [];
    const exclTags = hasExclTags ? envExclTags.split(' ') : [];

    let orgIt = it;
    let filterFunction = hasTags ? onlyWithTags : onlyWithoutTags;
    let filteredIt = filterFunction as TestFunction;
    filteredIt.skip = orgIt.skip;
    filteredIt.only = orgIt.only;
    filteredIt.retries = orgIt.retries;
    it = filteredIt;

    function onlyWithTags(title, fn) {
        if (tags.find(t => title.indexOf(t) === -1)) {
            fn = null;
        }
        orgIt(title, fn);
    }

    function onlyWithoutTags(title, fn) {
        if (exclTags.find(t => title.indexOf(t) !== -1)) {
            fn = null;
        }
        orgIt(title, fn);
    }
})()

Notice I ignore all tags if you try to both include and exclude. I found an issue, when using .skip or .only together with this code, but hasn't bothered to try an solve it. Use as is if you want, it's worked both with 4.1 and 4.2 😁

[coder-shanks]

@thviQit Thanks for providing the script. I just have a doubt if there is any effect of running the tests in headless or headed browser mode. Is anything prerequisite here?

[thviQit]

@coder-shanks Nope. Works with cypress run and cypress open. In headless it prints the skipped tests as pending.

[jennifer-shehane]

This issue is still in the 'ready for work' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

[UncleGus]

@thviQit does your solution only work with TypeScript? I am trying to avoid setting up my suite with Typescript, and I converted your code (just changed the import to require and dropped the as TestFunction) but it does not appear to be running when I execute tests.

If I leave it as a TypeScript file and try to import/require it in index.js, the tests can't find the module.

[thviQit]

@UncleGus We use gulp to compile the TypeScript into javascript, and the script is written to be used with types. Going to try the new functionality in Cypress 4.4, where it's not necessary to compile to javascript, it's going to be nice!

Anyway. The transpiled version looks like this:

// support/filterTestByTags.js
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
(function () {
    var _a, _b;
    if (!Cypress.env('tags') && !Cypress.env('exclTags')) {
        return;
    }
    var envTags = (_a = Cypress.env('tags')) !== null && _a !== void 0 ? _a : '';
    var envExclTags = (_b = Cypress.env('exclTags')) !== null && _b !== void 0 ? _b : '';
    var hasTags = envTags !== '';
    var hasExclTags = envExclTags !== '';
    // Don't filter if both is defined. We do not know what is right
    if (hasTags && hasExclTags) {
        console.log('Both tags and excluding tags has been defined. Not filtering');
        return;
    }
    var tags = hasTags ? envTags.split(' ') : [];
    var exclTags = hasExclTags ? envExclTags.split(' ') : [];
    var orgIt = it;
    var filterFunction = hasTags ? onlyWithTags : onlyWithoutTags;
    var filteredIt = filterFunction;
    filteredIt.skip = orgIt.skip;
    filteredIt.only = orgIt.only;
    filteredIt.retries = orgIt.retries;
    it = filteredIt;
    function onlyWithTags(title, fn) {
        if (tags.find(function (t) { return title.indexOf(t) === -1; })) {
            fn = null;
        }
        orgIt(title, fn);
    }
    function onlyWithoutTags(title, fn) {
        if (exclTags.find(function (t) { return title.indexOf(t) !== -1; })) {
            fn = null;
        }
        orgIt(title, fn);
    }
})();

// support/index.js
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
require("./commands");
require("./filterTestsByTags");

If you are trying to avoid the types, your code should probably look something like that. Remember to add the tags in the --env parameter when running Cypress as shown in https://github.com/cypress-io/cypress/issues/1865#issuecomment-601585432 or nothing happens.

[UncleGus]

Hmm, no luck with the transpiled code. It is still just executing all the tests. And switching back to Typescript in v4.4.1 still gets the same error: Error: Cannot find module '/filterTestsByTags' from '/home/.../cypress/support'

[UncleGus]

I've just converted my test spec file to Typescript, added the tsconfig.json and all that, and it's still not working. I must be missing something. I also have a number of errors/warnings in the filter file, because the two functions' parameters are not typed, but I don't think that's a significant issue.

[kirill-golovan]

@jennifer-shehane does Cypress dashboard count skipped tests as a record?

[KMKoushik]

@jennifer-shehane Is someone working on this feature?

[jennifer-shehane]

@kirill-golovan No, skipped tests are not counted as part of your Billing.

This issue is still in the 'proposal' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

[wilsoncd35]

Thanks for this. Sharing my use case, if it helps.

We tag our mocha tests that are proving known bugs. We want to keep the test but still allow the pipeline to pass.

# Run tests except known bugs and improvements.
mocha --grep '(@bug|@improve)' --invert test/foo.test.js

[jimpriest]

Ugh. Still no work on this? What do people do in CI to filter their tests? Really missing this after moving from Robot Framework :(

[x-yuri]

https://github.com/cypress-io/cypress/issues/1865#issuecomment-520179153 there are probably other solutions in the issue

[filiphric]

Ugh. Still no work on this? What do people do in CI to filter their tests? Really missing this after moving from Robot Framework :(

@jimpriest I have written up a solution that I use for grepping my tests, maybe it’ll help.

[jimpriest]

i may have found a solution using cypress-tags - https://github.com/annaet/cypress-tags

[meseguerfantasy]

any update @jennifer-shehane ?

[VickyLund]

This feature will add a lot of value to us. If we are able to add multiple tags then each of our team can slice the tests based on their requirements.

[midleman]

@thviQit thanks for sharing your example. it works for me... except if i have a before (or after) block in the spec file, it will still run the code in the before block even if all it blocks are being skipped (aka not tagged). is there a possible workaround/solution for this?

[marsimeau]

For anyone using @thviQit's workaround, if you want to use it.only and it.skip you have to rewrite those functions as they use the global it function internally, but it will break because it's not the original one.

After looking at Mocha's source, they're pretty straight forward to replicate:

filteredIt.skip = (title) => originalIt(title);
filteredIt.only = (title, fn) => {
  const test = originalIt(title, fn);
  test.parent.appendOnlyTest(test);
};

This has worked pretty well for me.