stryker-mutator / stryker-js

Mutation testing for JavaScript and friends
https://stryker-mutator.io
Apache License 2.0
2.59k stars 250 forks source link

Instrumentation fails for strings containing urls #2402

Closed Lakitna closed 4 years ago

Lakitna commented 4 years ago

Summary

This is a bug regarding Stryker 4 beta 3.

When instrumenting an object definition with as below:

const object = {
    foo: '',
    bar: 'file://',
};

Stryker throws the following error

12:46:51 (23900) ERROR DryRunExecutor One or more tests resulted in an error:
        SyntaxError: Invalid or unexpected token
{{projectRoot}}\.stryker-tmp\sandbox475018\src\{{sourceFile}}.js:32
  bar: __global_69fa48.__activeMutant__ === 685 ? "" : (__global_69fa48.__coverMutant__(685), 'file:
                                                                                              ^^^^^^

SyntaxError: Invalid or unexpected token
    at wrapSafe (internal/modules/cjs/loader.js:1117:16)
    at Module._compile (internal/modules/cjs/loader.js:1165:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1221:10)
    at Module.load (internal/modules/cjs/loader.js:1050:32)
    at Function.Module._load (internal/modules/cjs/loader.js:938:14)
    at Module.require (internal/modules/cjs/loader.js:1090:19)
    at require (internal/modules/cjs/helpers.js:75:18)
    at Object.<anonymous> ({{projectRoot}}\.stryker-tmp\sandbox475018\src\{{sourceFile}}.js:40:21)
    at Module._compile (internal/modules/cjs/loader.js:1201:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1221:10)

This only happens with a URL string (including protocol) that is preceded by an empty string. When the second string does not contain *:// this does not happen.

Additional note. It does not matter if the empty string is nested. The following throws the same error:

const object = {
    fizz: {
        buzz: '',
    },
    bar: 'file://',
};
const object = {
    fizz: [
        '',
    ],
    bar: 'file://',
};

Stryker config

module.exports = function(config) {
    config.set({
        // Empty strings in array due to https://github.com/stryker-mutator/stryker/issues/2305
        mutate: [
            'src/**/*.js',
            '!{src,lib}/**/*.+(spec|test).js?(x)',
            '',
            '',
        ],
        packageManager: 'npm',
        reporters: [
            'html',
            'progress',
        ],
        testRunner: 'mocha',
        mochaOptions: {
            spec: [
                './test/**/*.js',
                './src/**/*.test.js',
            ],
        },
        thresholds: {
            high: 80,
            low: 60,
            break: 50,
        },
    });
};

Test runner config

module.exports = {
    spec: [
        'test/**',
    ],
    reporter: 'mocha-multi-reporters',
    'reporter-option': [
        'configFile=.mocha-reporterrc.js'
    ],
    recursive: true,
    exclude: [
        '.git',
        'node_modules',
        'test/**/resources/**',
    ],
    ui: 'BDD-bundle',
    require: [
        'choma',
        'chai/register-expect',
        'mocha-bundle-ui',
        './test/setup.js',
    ],
    'watch-files': [
        'test/**/*',
        'src/**/*',
    ],
};

Stryker environment

    "@stryker-mutator/core": "^4.0.0-beta.3",
    "@stryker-mutator/javascript-mutator": "^3.3.1",
    "@stryker-mutator/mocha-runner": "^4.0.0-beta.3",
    "chai": "^4.2.0",
    "chai-arrays": "^2.0.0",
    "chai-as-promised": "^7.1.1",
    "chai-string": "^1.5.0",
    "choma": "^1.2.1",
    "mocha": "^8.0.1",
    "mocha-bundle-ui": "^3.0.0",
    "mocha-multi-reporters": "^1.1.7",
    "nyc": "^15.1.0",
    "proxyquire": "^2.1.3",
    "sinon": "^9.0.2",
    "sinon-chai": "^3.5.0"

Test runner environment

    "test": "mocha",
    "test:mutation": "stryker run .strykerrc.js",

Your Environment

software version(s)
node v14.5.0
npm 6.14.5
Operating System Windows 10 2004

Add stryker.log

stryker.log

nicojs commented 4 years ago

I'll have a look at this issue tomorrow

nicojs commented 4 years ago

I can't reproduce this. Running on express crashes with this error:

14:13:35 (26924) ERROR DryRunExecutor One or more tests resulted in an error:
        SyntaxError: Invalid or unexpected token
C:\z\github\stryker-mutator\stryker-2\perf\test\express\.stryker-tmp\sandbox4426232\test\app.router.js:690
      app.get('/file
              ^^^^^^

SyntaxError: Invalid or unexpected token
    at wrapSafe (internal/modules/cjs/loader.js:1072:16)
    at Module._compile (internal/modules/cjs/loader.js:1122:27)

If you then look at the file, (test/app.router.js), you'll see something interesting:

    it('should require a preceding /', function(done){
      var app = express();

      app.get('/file

There the test ends. End of file.

The problem is with the strip-comments dependency. I've created an issue: https://github.com/jonschlinkert/strip-comments/issues/60

Disabling stripping of comments solves it:

  "sandbox": {
    "stripComments": false
  }

We're using strip-comments to remove // @ts-expect-error or // @ts-check from test and production code files. This is needed for typescript compilation to work (we're adding // @ts-nocheck on top of each file, where your own // @ts-expect-error and // @ts-check might interfere with). More info: https://github.com/stryker-mutator/stryker/pull/2365

I was hoping that strip comments would (almost) always be safe to do. But I see it has some unfortunate side effects and now this bug. @brodybits isn't happy with this solution either (#2407).

Any ideas on how to solve this issue better would be much appreciated. I'll open a new discussion issue and closing this one for now.