UpdateOperatorMutator is causing Karma to disconnect

[CarlosRosario]

Hello,

I am using Stryker on an AngularJS project with the following npm modules:

• "grunt-stryker": "0.5.2",
• "stryker": "0.6.2",
• "stryker-api": "0.5.3",
• "stryker-html-reporter": "0.4.2",
• "stryker-jasmine": "0.2.2",
• "stryker-karma-runner": "0.4.1"

Whenever Stryker mutates native Javscript for/while loops that use increment/decrement operators, those mutations cause Karma to disconnect and mark all remaining mutations as 'survived'. The error message i see in the console looks like:

Mutation testing 31% (ETC 480s) [27 killed] [0 survived] [0 no coverage] [0 timeout] [0 error] [2017-08-11 15:09:58.742] [TRACE] IsolatedTestRunnerAdapter - 11 08 2017 15:09:58.741:INFO [framework.browserify]: bundle built

[2017-08-11 15:10:01.060] [TRACE] IsolatedTestRunnerAdapter - 11 08 2017 15:10:01.061:INFO [karma]: Restarting Chrome 56.0.2924 (Windows 10 0.0.0) (1 of 5 attempts)

[2017-08-11 15:10:01.068] [TRACE] IsolatedTestRunnerAdapter - [2017-08-11 15:10:01.068] [INFO] KarmaTestRunner - karma run done with 1

[2017-08-11 15:10:01.070] [DEBUG] EventRecorderReporter - Writing event onMutantTested to file reports\mutation\events\00030-onMutantTested.json [2017-08-11 15:10:01.078] [TRACE] IsolatedTestRunnerAdapter - No captured browser, open http:

[2017-08-11 15:10:01.079] [TRACE] IsolatedTestRunnerAdapter - [2017-08-11 15:10:01.079] [INFO] KarmaTestRunner - karma run done with 1

[2017-08-11 15:10:01.080] [DEBUG] EventRecorderReporter - Writing event onMutantTested to file reports\mutation\events\00031-onMutantTested.json [2017-08-11 15:10:01.087] [TRACE] IsolatedTestRunnerAdapter - No captured browser, open http://

[2017-08-11 15:10:01.088] [TRACE] IsolatedTestRunnerAdapter - [2017-08-11 15:10:01.089] [INFO] KarmaTestRunner - karma run done with 1

[2017-08-11 15:10:01.090] [DEBUG] EventRecorderReporter - Writing event onMutantTested to file reports\mutation\events\00032-onMutantTested.json [2017-08-11 15:10:01.097] [TRACE] IsolatedTestRunnerAdapter - No captured browser, open http://

These error messages repeat for a while and then jumps straight to 99% with the remaining mutations marked as 'survived' mutants.

I created a simple angular service and corresponding Jasmine spec file to make sure i could isolate and reproduce the issue i am seeing:

stryker_test.service.js

/*global angular, require*/
require('angular');
(function() {
    'use strict';
    angular.module('stryker_testModule', [])
     .factory('stryker_test', function() {
        var index,
            dataValue = 0,
            testFunction = function() {
                for(index = 0; index < 10; index++){
                    dataValue += 1;
                }
            },

            getDataValue = function() {
                return dataValue;
            };

        return {
            testFunction: testFunction,
            getDataValue: getDataValue
        };
    });
}());

stryker_test.service.spec.js

/*global angular, require, describe, it, afterAll, beforeAll, beforeEach, afterEach, module, expect, inject, spyOn  */
require('angular-mocks');
(function() {
    'use strict';

    describe('stryker test service: ', function() {
        angular.mock.module.sharedInjector();
        beforeAll(angular.mock.module('stryker_testModule'));

        beforeAll(inject(function($injector) {
            this.strykerTestService = $injector.get('stryker_test');
        }));

        afterAll(angular.mock.module('stryker_testModule'));

        describe('When stryker test function is called', function() {

            beforeEach(function() {
                this.strykerTestService.testFunction();
            });

            it('should give the datavalue correct value', function() {
                expect(this.strykerTestService.getDataValue()).toEqual(10);
            });
        });
    });
}());

So far i am able to get around this issue by commenting out the code inside of applyMutations() in the UpdateOperatorMutator.js code:

stryker/src/mutators/UpdateOperatorMutator.js

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
var esprima_1 = require("esprima");
var UpdateOperatorMutator = (function () {
    function UpdateOperatorMutator() {
        this.name = 'UpdateOperator';
        this.type = esprima_1.Syntax.UpdateExpression;
        this.operators = {
            '++': '--',
            '--': '++'
        };
    }
    UpdateOperatorMutator.prototype.applyMutations = function (node, copy) {
        //if (node.type === this.type && this.operators[node.operator]) {
        //    var mutatedNode = copy(node);
        //    mutatedNode.operator = this.operators[node.operator];
        //    return mutatedNode;
        //}
    };
    return UpdateOperatorMutator;
}());
exports.default = UpdateOperatorMutator;
//# sourceMappingURL=UpdateOperatorMutator.js.map

If it helps debug, i am using angular v1.4.7 and Jasmine v2.5.2. I'm not sure if this is a Stryker issue or a Karma issue.

Edit: I also saw this issue and thought it sounded kind of similar to what i am seeing. I'm not using karma-nightmare though.

Edit 2: Changing my for loop to use index+=1 instead of index++ also gets rid of the issue, but i'm pretty sure it's just sneaking past Stryker so i never get into infinite loop scenarios during mutation testing.

[nicojs]

@CarlosRosario thanks for reporting this issue.

It seems that the mutant is causing an error of which Stryker is not aware. Causing browsers to disconnect from karma which goes unnoticed by stryker.

Could you please provide more information? What does your karma.conf and stryker.conf look like. What karma plugins do you have installed.

Even better would be a small github project that reproduces this issue. If you set maxConcurrentTestRunners to 1, you should be able to reliably reproduce the issue.

[CarlosRosario]

Hi @nicojs ,

Thanks for the quick response. As far as karma plugins i'm using:

• "grunt-karma": "2.0.0",
• "karma": "1.4.1",
• "karma-browserify": "5.1.0",
• "karma-chrome-launcher": "2.0.0",
• "karma-jasmine": "0.3.5",
• "karma-webdriver-launcher": "1.0.4"

My Stryker configuration looks like:

stryker.conf.js

config.set({
        logLevel: 'all',
        files: [
            { pattern: '/path/to/jsFile.js', mutated: true, included: true},
            '/path/to/jsFile.spec.js',
            { pattern: '/path/to/otherNecessaryFiles.js', mutated: false, included: true},
        ],
        testRunner: 'karma',
        testFramework: 'jasmine',
        coverageAnalysis: 'perTest',
        reporter: ['html', 'progress', 'event-recorder'],
        maxConcurrentTestRunners: 6,
        timeoutMs: 240000,
        karmaConfig: {
            singleRun: false,
            browsers: ['chrome'],
            preprocessors: {
                '/path/to/otherNecessaryFiles.js': ['browserify'],
                '/path/to/jsFile.spec.js': ['browserify']
            },
            frameworks: ['browserify', 'jasmine'],

            // I've toyed around with different values for these configurations. I think the ones here are default values, but i'm not too sure.
            retryLimit: 3,
            browserDisconnectTolerance: 3,
            browserNoActivityTimeout: 100000,
            captureTimeout: 180000
        }
    });

I'll work on creating a sample angular project that reproduces this issue. What did you mean by your last comment? I'm currently seeing the issue no matter what i've set maxConcurrentTestRunners to.

Small Edit: I've also tried using the browserDisconnectTimeout karma configuration property, and it seemed to help, but not consistently.

[CarlosRosario]

Here's a sample angular project that reproduces the issue with 1 maxConcurrentTestRunner

https://github.com/CarlosRosario/StrykerInfiniteLoopDemoProject

You should just have to do an npm install and be able to run stryker right away after that. I wasn't able to reproduce the issue happening for more than one concurrent test runner.. i was thinking maybe it's because only a small number of mutants are generated for the small code i wrote.

[nicojs]

Hi Carlos, thanks for all the trouble. I think i found the issue.

TL;DR;

We should never let karma handle timeouts. As a work around: remove timeoutMs: 240000 from your stryker config will force Stryker to handle timeouts.

What's wrong

The mutant is causing an infinite loop, which is not a problem normally for Stryker. It will kill the child process and spawn a new one, marking the mutant as a timeout. The problem is that not stryker, but Karma is reporting the timeout.

Karma itself has a mechanism of reporting timeouts when it hasn't heard from the browser in x time. We assumed that the karma process would still be responsive afterwards, but it is not (never assume anything). I think we should fix this by forcing a timeout to be handled by Stryker.

[CarlosRosario]

Just out of curiosity, how come the bug doesn't manifest when there are more than 1 concurrent runners? I tried forcing a couple hundred mutations (just having a function that returned a number multiplied by 1 over and over again).. but i was never able to reproduce the issue with more than 1 concurrent test runner.

[simondel]

@nicojs Is this still a thing or has it been fixed since?

[simondel]

As we haven't heard anything about this so far, I'm closing this issue. If it's still a problem, let us know!