Babel v6 support

[kamesh95]

Question

Can I use stryker with Babel v6? I see that most of the dependencies in stryker use Babel v7 but we are still using Babel v6 in our react project. And when I try to use stryker with that project, it fails. So is there any configuration that I can use to get stryker working on our project as well? Thanks.

Stryker environment

+-- @stryker-mutator/core@4.3.1
| +-- @stryker-mutator/api@4.3.1
| +-- @stryker-mutator/instrumenter@4.3.1
| | +-- @stryker-mutator/api@4.3.1 deduped
| | +-- @stryker-mutator/util@4.3.1 deduped
| +-- @stryker-mutator/util@4.3.1

+-- jest@26.6.3

Additional context

I tried installing lower jest versions that used babel v6 but still can't get it working because there are babel parser related issues everywhere.

[nicojs]

Hi @kamesh95 👋. Welcome to Stryker.

This scenario should be supported since Stryker brings its own version of babel. Stryker uses babel for its JS parser. However, I think the problem here is that babel will assume it can read the .babelrc file as a babel 7 config file.

This should be a way to confirm this. Please open your node_modules/@stryker-mutator/instrumenter/dist/src/parsers/js-parser.js file and change line 38-40:

        const ast = await core_1.parseAsync(text, {
            parserOpts: {
                plugins: [...(pluginsOverride !== null && pluginsOverride !== void 0 ? pluginsOverride : defaultPlugins)],
            },
            filename: fileName,
            sourceType: 'module',
+           babelrc: false            
        });

Try to see if you make it work this way.

Anyway: I think its smart for you to migrate to babel 7. Babel 6 is pretty much a dinosaur at this point.

[kamesh95]

Thanks @nicojs, I have tried this by adding babelrc: false and adding my project specific babel plugins along with the default plugins in the above file. Seems to be working fine with a subset of tests. But when I run it on my entire project with 375 test suites and around 8000+ test cases, it seems to be very slow.

Mutation testing 0% (elapsed: ~9m, remaining: ~2102h 55m) 4/54863 tested (3 survived, 0 timed out)

• Do you have any recommendations to speed up the process? I am using Jest with --runInBand config to avoid running into caching issues with Jest.
• When I use stryker + jest with --coverage argument, the entire process fails due to heap out of memory. But the same works with --coverage argument when I run the test cases using npm test.

My npm test command: "test": "node --max_old_space_size=4096 ./node_modules/jest/bin/jest.js --silent --logHeapUsage --colors --runInBand"

My Stryker config:

{
  concurrency: 4,
  packageManager: 'npm',
  dryRunTimeoutMinutes: 30,
  coverageAnalysis: 'off',
  reporters: ["html", "clear-text", "progress" ],
  testRunner: "command"
}

[nicojs]

Ok, a couple of things:

You have a large project. You might need to split up mutation testing over a couple of runs. You can do that using the "mutate" property for that.

You probably want to use the @stryker-mutator/jest-runner plugin. Plugging into the test runner is generally faster than using the command test runner. Since Stryker 4.3, the @stryker-mutator/jest-runner supports coverage analysis, so after switching you probably want to enable that as well: coverageAnalysis: "perTest". Note: Stryker uses its own coverage analysis and the jest-runner plugin disables jest's own coverage to improve the speed. It will also use runInBand and a couple of other settings to improve mutation testing, you won't have to worry about that.

Since your tests require a lot of memory, it might be best to limit concurrency even further, for example: concurrency: 2,. If test runner processes run out of memory, try to specify a custom --max_old_space_size using testRunnerNodeArgs

New config example:

{
  mutate: ['src/core/**/*.js'] // if needed
  concurrency: 2,
  packageManager: 'npm',
  dryRunTimeoutMinutes: 30,
  coverageAnalysis: 'perTest',
  reporters: ["html", "clear-text", "progress" ],
  testRunner: "jest",
  testRunnerNodeArgs: ['--max_old_space_size=4096'] // if needed
}

With this new config, your initial test run will still take long, since Stryker will still run all your unit tests once to measure mutation coverage per test. During mutation testing, it will only run the subset of tests that cover a specific mutant, so those runs will go a lot faster.

[nicojs]

For the babelrc: false thing. I propose adding a feature:

{
  mutator: {
    parserOptions: {  babelrc: false /*,  [...] */ }
  }
}

The mutator.parserOptions would simply be spreaded into the babel.parse() call.

Would that be enough to support your use case?

[kamesh95]

Update with below configuration:-

{
  concurrency: 2,
  packageManager: 'npm',
  dryRunTimeoutMinutes: 30,
  tempDirName: 'stryker-tmp',
  coverageAnalysis: 'perTest',
  reporters: ["html", "clear-text", "progress" ],
  testRunner: "jest",
  logLevel: 'trace'
}

Mutation testing 0% (elapsed: ~57m, remaining: ~7054h 50m) 7/51241 tested (1 survived, 0 timed out)

I used the jest-runner with concurrency 2 and now there are 51241 mutants, earlier it was 54863. So there seems to be some optimization. But I am trying to understand the behavior here, for each mutant I can see the entire test suites (npm test / jest) are executed again which means for 51241 mutants, Stryker will spawn 51241 test runners, 2 at a time. Since one entire test run takes around 17 mins for me, this would mean the total execution time to compute mutation score would be - (51241 x 17) / 2 [concurrency] minutes? Is it so?

If this is true, I was wondering why do we need to run all tests for each mutant? Can't we just run the specific ones related to the file in which the mutant exists?

[nicojs]

now there are 51241 mutants, earlier it was 54863

Coverage analysis is at play here. 54863 - 51241 = 3622 mutants were found without coverage and thus won't be tested (they will be reported as "NoCoverage")

But I am trying to understand the behavior here, for each mutant I can see the entire test suites (npm test / jest) are executed again.

No, not exactly. Or at least, that shouldn't be happening. How did you figure out that this is the case?

Stryker determines the test coverage per mutant in the initial test run. Then, during mutation testing, it will only run those tests that cover a specific mutant.

However, there is a thing we call static mutants which could be a factor here. A static mutant is a mutant that isn't covered by a test directly but instead is covered during the loading of a file.

For example:

// greeting.js
const GREETING = 'Hi '; // 1 Static mutant ('Hi ' -> '')!
export function greeting(name) {
  return GREETING + name; // 2. Normal (non-static) mutant (+ -> -)
}

// greeting.spec.js
import greeting from './greeting.js';
describe('greeting', () => {
  it('should say Hi Scott', () => {
    expect(greeting('Scott')).toBe('Hi Scott');
  });
});

In this case, the string 'Hi ' is read when the file is loaded (Not during the test run). This means that Stryker will default to running all tests, even the ones that have nothing to do with greeting. The second mutant is covered during the running of the first test, which means that Stryker will know to only run that one test when testing mutant 2.

One more caveat is that the @stryker-mutator/jest-runner will run with --findRelatedTests by default, so Jest should at least not run any other test files than the ones that import the file under test, even for static mutants.

[kamesh95]

@nicojs Thanks for the explanation. I checked running jest normally with --findRelatedTests on my project and that seems to be the cause for this behavior, it returns 250-300 test suites for each file out of total 375 test suites. This explains why it is taking so much time. I read this https://stackoverflow.com/questions/44066996/how-does-jest-findrelatedtests-work-under-the-hood and seems like it runs all dependent test suites transitively that we import in our file. I have last few questions:

1. Do you think it's a valid idea in our case to run single test suite via Stryker and mutate only the main file linked to it. Eg with command runner would look like:

   testRunner: "command",
   commandRunner : {
   command: 'npm test fileSpec.jsx'
   },
   mutate: ['file.jsx']

2. Like testRunnerNodeArgs, is there any option to pass jest related args as well with jest-runner? Like --silent to disable all test related logs or pass a filename so as to run selective tests?
3. Can we also add something to supply presets and plugins to babel config for stryker? Like adding react preset and some project specific babel plugins?

[kamesh95]

@nicojs Do you have an update on the above? :)

[nicojs]

Ow sorry, I lost track of this issue!

1. Do you think it's a valid idea in our case to run single test suite via Stryker and mutate only the main file linked to it. [...]

Sure, why not. Whatever is fastest for your use case, right. You will have to merge the reports together. I want to add this feature to mutation-testing-metrics, see https://github.com/stryker-mutator/mutation-testing-elements/issues/1180.

2. Like testRunnerNodeArgs, is there any option to pass jest related args as well with jest-runner

No, this is currently not possible, but I'm open to the idea. Feel free to send a PR our way. See https://stryker-mutator.io/docs/stryker-js/jest-runner#configuration for the config options we currently support.

3. Can we also add something to supply presets and plugins to babel config for stryker

No, currently not, but this should also be a minor addition. Again, feel free to send a PR our way.

So... I'm open to all 3 but work needs to be done to help you here.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.