search

karma-runner / karma-chrome-launcher

A Karma plugin. Launcher for Chrome and Chrome Canary.
MIT License
469 stars 119 forks source link

Chrome(Headless) randomly not running all tests and no errors reported. #213

Open activist opened 5 years ago

activist commented 5 years ago

I have an issue where we have a set of tests that randomly does not run in Chrome/ChromeHeadless with no errors reported. It is hard to give specific number, but 50% of the time karma quits too early. It is just like karma is stopping when it feels like doing so.

When running with Firefox/FirefoxHeadless it always completes all off the tests.

Environment: Ubuntu (18.04/19.10) Chrome version: 77.0.3865 Karma version: 4.3.0 karma-chrome-launcher@3.1.0

Example output:

Chrome 77.0.3865 (Linux 0.0.0): Executed 256 of 278 SUCCESS (0 secs / 2 mins 22.139 secs)
Chrome 77.0.3865 (Linux 0.0.0): Executed 256 of 278 SUCCESS (2 mins 34.995 secs / 2 mins 22.139 secs)
18 09 2019 13:05:31.719:DEBUG [launcher]: CAPTURED -> BEING_KILLED
18 09 2019 13:05:31.719:DEBUG [launcher]: BEING_KILLED -> BEING_FORCE_KILLED
TOTAL: 256 SUCCESS
18 09 2019 13:05:31.720:DEBUG [karma-server]: Run complete, exiting.
18 09 2019 13:05:31.720:DEBUG [launcher]: Disconnecting all browsers
18 09 2019 13:05:31.720:DEBUG [launcher]: BEING_FORCE_KILLED -> BEING_FORCE_KILLED
18 09 2019 13:05:31.788:DEBUG [launcher]: Process Chrome exited with code 0 and signal null
18 09 2019 13:05:31.788:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-20890304
18 09 2019 13:05:31.847:DEBUG [launcher]: Finished all browsers
18 09 2019 13:05:31.847:DEBUG [launcher]: BEING_FORCE_KILLED -> FINISHED
18 09 2019 13:05:31.848:DEBUG [launcher]: FINISHED -> FINISHED

Here we would expect 278 tests to run, but only 256 runs. Sometimes the number is far lower, so it is also seemingly random at which test karma feels like stopping and which tests have ran.

Here is the main part of karma-conf.js:

...
        reporters: ['imagecapture', 'progress', 'json-log'],
        port: 9876,  // karma web server port
        colors: true,
        logLevel: config.LOG_INFO,
        browsers: browsers,
        autoWatch: false,
        singleRun: true,
        concurrency: Infinity,
        plugins: [
            'karma-*',
            require('./karma-imagecapture-reporter.js')
        ],
        sharding: {
          specMatcher: /(spec|test|demo)s?\.js/i
        },
        jsonLogReporter: {
            outputPath: 'test/',
        },
        customLaunchers: {
            ChromeCI: {
                base: 'Chrome',
                flags: ['--no-sandbox']
            },
        },
...

I might be relevant to know that we use or own karma-imagecapture-reporter to store images from svg/dom nodes and then later do a pixel-by-pixel comparison. It looks like this:

const fs = require('fs');

function ImageCaptureReporter(
    emitter
) {

    function prettyXML(svg) {
        svg = svg
            .replace(/>/g, '>\n')
            .replace(/<tspan([^>]*)>\n/g, '<tspan$1>')
            .replace(/<\/tspan>\n/g, '</tspan>');

        return svg;
    }

    emitter.on('browser_info', (browser, info) => {
        let data = info.data;
        let filename = info.filename;
        if (/\.svg$/.test(filename)) {
            fs.writeFileSync(filename, prettyXML(data));
        } else if (/\.png$/.test(filename)) {
            data = data.replace(/^data:image\/\w+;base64,/, '');
            let buf = new Buffer(data, 'base64');
            fs.writeFileSync(filename, buf);
        } 
    });
}

ImageCaptureReporter.$inject = [
    'emitter'
];

module.exports = {
    'reporter:imagecapture': ['type', ImageCaptureReporter]
};

Do you have any clues as to what the issue could be?

johnjbarton commented 5 years ago

Do you have any clues as to what the issue could be?

In my experience the most likely cause is an error in async app or test code. For example a missing await on a promise. Maybe in beforeEach() code in one describe. This causes a race between the promise and the overall test run exiting. If the promise resolves, the test passes; if not the test exists with not enough results. Timing on different machines and different browsers give different results.

kdevcse commented 4 years ago

Seeing this issue too.. it's the most bizarre thing.

fatihdestegul commented 4 years ago

It is an old issue but did you try to increase following configs inside customLaunchers ?

Seems karma-launcer sends a SIGKILL because it exceeds to threshold.. @activist