Same configuration in different react apps is causing failure in mutation tests at all or finds just few tests

[Vaclovas]

Question

Why same configuration for a very similar app is causing test failure? Mainly, tests are not even running or only a few tests can be found.

Stryker environment

stryker config: { "reporters": [ "html", "clear-text", "progress" ], "packageManager": "npm", "timeoutMS": 30000, "ignorePatterns": ["/src/lib/icons/*.tsx"], "tsconfigFile": "tsconfig.json", "mutator": { "excludedMutations": ["BlockStatement", "StringLiteral", "ArrayDeclaration", "ObjectLiteral"] }, "disableTypeChecks": "*/.{js,ts,jsx,tsx,html,vue}", "testRunner": "jest", "coverageAnalysis": "perTest", "tempDirName": "stryker-tmp", "mutate": [ "/src///*.ts", "/src///.tsx", "!/src//.spec.ts", "!/src/*/.spec.tsx" ] }

nx + jest+ react and "@stryker-mutator/core": "8.2.6", "@stryker-mutator/jest-runner": "8.2.6", "jest": "29.4.1", "jest-environment-jsdom": "29.4.1", "jest-environment-node": "29.4.1", "jest-slow-test-reporter": "1.0.0",

Additional context

Here is the first app:

here is another app: What is intresting, that even components are covered by unit tests, they are marked as "NoCoverage", jest finds these tests and succesfully adds those to coverage report, but stryker - is failing. Assumption can be, that configuration is bad, but some of the files are found and counted into the report: So, actually, only 6 spec.ts/tsx files are identified from 256 files. I've tried to search for differences, remove specific imports, change naming - nothing happens, tried some recipies from github and troubleshooting guide.

Maybe you could assist with this?

[Vaclovas]

ok, seems like I've found the issue which is skipoing those tests - this is aliased imports... Can you provide info, why those are not working?

[nicojs]

Hmm interesting. I think this is an issue with Jest. Could you try running Stryker with this change in your config?

{
  "reporters": ["html", "clear-text", "progress"],
  "packageManager": "npm",
  "timeoutMS": 30000,
  "ignorePatterns": ["/src/lib/icons/*.tsx"],
  "tsconfigFile": "tsconfig.json",
  "mutator": { "excludedMutations": ["BlockStatement", "StringLiteral", "ArrayDeclaration", "ObjectLiteral"] },
  "disableTypeChecks": "/.{js,ts,jsx,tsx,html,vue}",
  "testRunner": "jest",
  "coverageAnalysis": "perTest",
  "tempDirName": "stryker-tmp",
-  "mutate": ["/src//**/.ts", "/src///*.tsx", "!/src//*.spec.ts", "!/src/**/*.spec.tsx"]
+  "mutate": ["/src//**/.ts", "/src///*.tsx", "!/src//*.spec.ts", "!/src/**/*.spec.tsx"],
+  "jest": {
+    "enableFindRelatedTests": false
+  }
}

[Vaclovas]

Hmm interesting. I think this is an issue with Jest. Could you try running Stryker with this change in your config?

{
  "reporters": ["html", "clear-text", "progress"],
  "packageManager": "npm",
  "timeoutMS": 30000,
  "ignorePatterns": ["/src/lib/icons/*.tsx"],
  "tsconfigFile": "tsconfig.json",
  "mutator": { "excludedMutations": ["BlockStatement", "StringLiteral", "ArrayDeclaration", "ObjectLiteral"] },
  "disableTypeChecks": "/.{js,ts,jsx,tsx,html,vue}",
  "testRunner": "jest",
  "coverageAnalysis": "perTest",
  "tempDirName": "stryker-tmp",
-  "mutate": ["/src//**/.ts", "/src///*.tsx", "!/src//*.spec.ts", "!/src/**/*.spec.tsx"]
+  "mutate": ["/src//**/.ts", "/src///*.tsx", "!/src//*.spec.ts", "!/src/**/*.spec.tsx"],
+  "jest": {
+    "enableFindRelatedTests": false
+  }
}

1. Jest is running successfully and tests are being identified.
2. After the configuration change, all tests can be seen in the tests tab, but all cases survived, none were identified as killed.
3. If I'm changing aliased import to relate one - Stryker is able to find a test and killed cases ar identified.
4. 4. Running time increased by a huge amount.

[nicojs]

This sounds like this known issue: https://stryker-mutator.io/docs/stryker-js/troubleshooting/#all-mutants-survive---jest-runner

Could you try enabling --inPlace? That should do the trick. Note: using --inPlace doesn't allow you to change your code when StrykerJS is running.

[Vaclovas]

Hello, --inPlace is not changing any results. I've created a project boilerplate for this: project.zip open project, run "npm i" inside the root, after that continue the same on shell and "ui-shared" folders. Run "npx stryker run" with changing FieldLabel.spec.tsx file. Comment/uncomment import line for the same file import, and see how results are changing:

report difference:

[nicojs]

Hi @Vaclovas thanks for your detailed report.

The problem is that StrykerJS will try to use Jest's --findRelatedTests feature by default. This will make sure only the tests that actually cover files with mutants are run (a performance optimization). However, this doesn't work with the '@scp import.

You can verify this by running npx jest --findRelatedTests yourself:

nicojs@NicoJ01 project/ui-shared (master)$ npx jest --findRelatedTests src/lib/field-label/FieldLabel.tsx 
No tests found, exiting with code 1
Run with `--passWithNoTests` to exit with code 0
No files found in ~/tmp/project/ui-shared.
Make sure Jest's configuration does not exclude this directory.
To set up Jest, make sure a package.json file exists.
Jest Documentation: https://jestjs.io/docs/configuration
Pattern: src/lib/field-label/FieldLabel.tsx - 0 matches

There are 2 ways to solve this:

1. Disable findRelatedTests for StrykerJS. This comes with a performance penalty:

   {
     "$schema": "../node_modules/@stryker-mutator/core/schema/stryker-schema.json",
     "reporters": [
       "html", "clear-text", "progress"
     ],
     "packageManager": "npm",
     "timeoutMS": 30000,
     "ignorePatterns": ["**/src/lib/icons/*.tsx"],
     "tsconfigFile": "tsconfig.json",
     "mutator": { "excludedMutations": ["StringLiteral", "BooleanLiteral", "ObjectLiteral"] },
     "disableTypeChecks": "**/*.{js,ts,jsx,tsx,html,vue}",
     "testRunner": "jest",
     "coverageAnalysis": "perTest",
   - "tempDirName": "stryker-tmp"
   + "tempDirName": "stryker-tmp",
   + "jest": {
   +   "enableFindRelatedTests": false
   + }

2. Configure moduleNameMapping in ui-shared/jest.config.ts.

   /* eslint-disable */
   export default {
     displayName: 'ui-shared',
     preset: './jest.preset.js',
     transform: {
       '^(?!.*\\.(js|jsx|ts|tsx|css|json)$)': '@nx/react/plugins/jest',
       '^.+\\.[tj]sx?$': ['babel-jest', { presets: ['@nx/react/babel'] }],
     },
     moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx'],
     coverageDirectory: '../coverage/ui-shared',
     testEnvironment: 'jsdom',
   - setupFilesAfterEnv: ['<rootDir>/testSetup.ts']
   + setupFilesAfterEnv: ['<rootDir>/testSetup.ts'],
   + moduleNameMapper: {
   +  '^@scp/(.*)$': '<rootDir>/src/$1',
   + },
   };

After choosing one of these 2 solutions, you can run stryker without --inPlace