Dynamically filter tests to run

[nicojs]

Is there a way to dynamically filter tests to run based on index / test name? I know you can already skip a test, but i'm looking for a way to dynamically decide which tests to skip, like this jasmine function: jasmineEnv.specFilter

If it is not available, would you be open to PR's? Possible this could be available to the testStart callback? For example: return 'skip' if the test needs to be skipped?

Background:

We're thinking of adding support for qunit to the stryker mutation testing framework. Having this feature will improve the performance of mutation testing with qunit test suites.

[trentmwillis]

You can configure the QUnit.config.filter and QUnit.config.testId properties from within the JS. These are checked before running each test to verify whether a test is supposed to run or not.

[datokrat]

These filters decide based on the the test description (which also provides the testId after being hashed), as far as I understand it. Is it also possible to use other criteria, for example to run the fifth and sixth test (based on their position)?

[trentmwillis]

Currently, ids or description are the only ways to filter. Since we support test reordering, indices aren't particularly useful as they can wind up being different across runs.

[datokrat]

Hmm, I see. So there is also no "initial ordering" preserved?

[trentmwillis]

You can disable reordering via QUnit.config.reorder.

Perhaps an explanation of your use case would help us figure out what the best way to accomplish it would be.

[datokrat]

Thank you for your quick response.

It's also about Stryker: To verify that tests cover all functionality, small changes to the source code are inserted into the code. The tests should catch these errors. Therefore, for every mutation, all tests are run - this can be quite slow.

But with coverage analysis, we can find out which tests don't even cover the mutated code and skip them. Maybe @nicojs can help here, but as I understand it, Stryker needs the index of the executed test.

QUnit.config.reorder is an interesting point. I'll figure out whether it fits our needs!

[nicojs]

Stryker needs the index of the executed test.

This is the current implementation, correct.

You can configure the QUnit.config.filter

Could we allow to provide a custom filter function here? That way, it doesn't matter what we use, test ids, test names, position of the stars,...

[mike-north]

@trentmwillis, you mentioned

You can configure the QUnit.config.filter and QUnit.config.testId properties from within the JS. These are checked before running each test to verify whether a test is supposed to run or not.

Can you point me to where this is happening? It seems like once the test is added to the processing queue, it's going to run regardles of filters, testId, moduleId, etc... If both QUnit.config.moduleId and QUnit.config.testId behaved the way you say, it would unlock my ability to do some really cool stuff with training/tutorial content.

My use cases involve specifying which tests to run (by testId is fine) AFTER the modules/tests are defined and BEFORE any of them have actually started to run.

[trentmwillis]

@mike-north sorry, looks like I was mistaken. The validation check is applied when the test is queued: https://github.com/qunitjs/qunit/blob/5fbaa489367e9bc0eac19d335aed1058920ffae5/src/test.js#L377-L379

[mike-north]

How big an ask would it be to check the testId filter before the tests are run? As it stands, by the time I know about which tests are available, I have already lost the opportunity to define a useful filter.

[trentmwillis]

I don't think it should be too hard to make that change (given that's how I thought it worked previously).

[Krinkle]

@nicojs I see a few mentions of QUnit in the Stryker org. Did you get this to work or is there something we can help with?

It you'd looking for a way to fully hook into the tests as they are being registered, I would actually recommend overloading the QUnit.test function. That would give you the function object by reference (if that's useful) and the test name and order etc.

If not, let me know what you need to decide whether a test should run.

[Krinkle]

🤖 Closing stale issue. If this is still an issue, feel free to mention me here, or create a new issue. You can also chat with us!

[NahueBerg]

Hi @Krinkle, I ended up reading this issue because I'm looking for a way to skip certain tests before execution based on a list of test names I'm getting in the beforeEach() function. As it's been a long time since this issue was discussed, I want to know if there is a way to filter those tests out without having to override the QUnit.test function. I thought about using QUnit.config.filter with a regex composed by all the test names concatenated in a string with '|' between each one of them, but I don't know if there's a maximum length supported for the regex or if it could potentially cause any issues in the future if the list is too long (i.e.: 100 tests). Thanks in advance!

[ro0gr]

@NahueBerg I think you can label tests that you want to skip with a tag-like substring, like:

test('[skip certain test] ...', function(...

and then use an inversed filter, like

QUnit.config.filter =  '![skip certain test]';

[NahueBerg]

@ro0gr that's what I meant in the part where I mentioned using a regex. The issue with that is that I don't know which tests I'm going to skip, the list of tests could be way too long to have a filter with all test names concatenated. Imagine if I had to skip 2000 tests, having 2000 test names concatenated in a regex string to filter out the tests doesn't sound good at all.

[Krinkle]

@NahueBerg I can probably suggest a number of possible solutions, but I need a bit more context I think. What is the end-user developer's reason for skipping certain tests? How do you envision them controlling this end-to-end, i.e. what is the path upto to the beforeEach() where you are currently trying to implement this?

[NahueBerg]

Hi @Krinkle, thanks for answering! We have a list of tests that should be skipped that changes based on different events through the day. We retrieve this list from a GET endpoint, and with that response we want to filter out the tests that should be skipped in that run and only for that run. The fact that the list is in constant change makes it impossible to know beforehand which tests will have to be skipped, so it's something we have to determine right before the start of the run. I supposed there was a way of doing something similar to the regex in the QUnit.config.filter object, but instead of a regex it would be a lambda that allowed me to look for the test name inside the list of tests to skip, but I don't see anything like that in the documentation. That's the reason why I thought about overriding the QUnit.test method to perform that check and then deciding whether I want to skip the test or not, something like this:

var originalTest = QUnit.test; QUnit.test = function(name, callback) { // Check if the test name is in the list of tests to skip if (testsToSkip.indexOf(name) !== -1) { QUnit.skip(name, callback); // Skip the test } else { originalTest.apply(this, arguments); // Call the original QUnit.test method } };

I'm about to try it to see if it works. If it does, would it be the best way to achieve what I want, or is there a more efficient way to do so?

[NahueBerg]

Hi again @Krinkle. I tried the solution above but I'm experiencing a weird issue. If I run, let's say, 4 tests, they pass but a fifth one gets executed too; that test doesn't have name, doesn't do anything, and its execution always times out. Do you know if this is a known issue when overriding the QUnit.test function?

[Krinkle]

@NahueBerg Can you obtain a stack trace from your custom function call? E.g. console.warn(new Error().stack).

As for known issues, not that I know, however, I can't say it's supported either. Overriding the method definition in this way, will, for example, cause QUnit.test.each to become undefined. Internally generated tests may also bypass your override, but that's probably a good thing in your case. E.g. you wouldn't want to supress reporting of uncaught errors, or the "No tests found" message and such.

[NahueBerg]

@Krinkle

Overriding the method definition in this way, will, for example, cause QUnit.test.each to become undefined.

That's exactly what was happening, which I fixed saving the original QUnit.test object into a variable called originalTest, and then doing:

QUnit.test.only = function (name, dataset, callback) { originalTest.only(name, dataset, callback); };

QUnit.test.skip = function (name, callback) { originalTest.skip(name, callback); };

QUnit.test.todo = function (name, callback) { originalTest.todo(name, callback); };

QUnit.test.each = function (name, dataset, callback) { originalTest.each(name, dataset, callback); };

That way I don't have troubles anymore with those methods but I'm surely missing many other properties from the original QUnit.test; since I don't use them now it's not an issue, but if I wanted to use any of them in the future I would need to add them to the new QUnit.test I've overrode, which isn't something I'm happy with. This worked for me, it's not what I would have expected to have to do but it works. If there was a way to filter tests using a lambda function or something like that it would be awesome, please consider it for future releases. Thank you for your help!!

[NullVoxPopuli]

I found that I can do this without overriding any of QUnit's APIs (which I expect would become read-only in the future, especially as that is the default behavior of modules if/when QUnit ships pure ESM)

https://jsbin.com/jigimaq/edit?html,output

code:

import QUnit from 'qunit';
import { setup } from 'qunit-dom';

setup(QUnit.assert);

QUnit.config.autostart = false;

// this is needed because QUnit didn't properly configure their exports.
// they _only_ have a default export.
const { module, test, skip } = QUnit;

module('example', function() {
  test('it works', function(assert) {
    assert.true(true);
    assert.dom('button').exists();
  });

  test('it is skipped via external means', function(assert) {
    assert.true(true);
    assert.dom('button').exists();
  });

  skip('always skipped', function (assert) {});

  module('nested', function() {
    test('a nested test', function (assert) {
      assert.true(true); 
    })
  });
});

// List of tests:
console.log(QUnit, QUnit === QUnit.QUnit)

function getTests() {
  function getTestsForModule(mod) {
     if (!mod) return [];

     let tests = mod.tests.map(x => ({ moduleName: mod.name, test: x }));
     return tests;
  }

  let result = QUnit.config.modules.map(getTestsForModule).flat().filter(Boolean);

  return result;
}

let allTests = getTests();

const toSkip = [
  'it is skipped via external means',
];
const skippedTests = new Set();

allTests.forEach(info => {
  let fullName = `${info.moduleName} > ${info.test.name}`;

  let shouldSkip = toSkip.some(x => fullName.includes(x));

  console.log(fullName, shouldSkip);
  if (shouldSkip) {
    skippedTests.add(info.test.testId);
  }
});

QUnit.hooks.beforeEach(function (assert) {
  let id = assert.test.testId;

  console.log(assert.test);

  if (skippedTests.has(id)) {
    assert.test.skip = true;
    assert.test.testReport.skipped = true;
  }
})

QUnit.start();