Incorrect Karma Browser.state: on second run tests hang forever with "Executed 0 of XXX ...".

solo-md commented 6 years ago

Expected behaviour

Tests should run. Once a watched file is changed, tests should re-run.

Actual behaviour

First test runs perfectly fine. After that Browser.state (karma/lib/browser.js) is READY. Once a watched file is changed, tests re-run. But there is nothing that changes the browser state to EXECUTING, so once a spec is completed, the Browser's onResult is ignored (the check is commented with // ignore - probably results from last run (after server disconnecting))

Environment Details

Karma version (output of karma --version): 2.0.4
Relevant part of your karma.config.js file

{
    // custom karma plugins
    plugins: [
      "karma-chrome-launcher",
      "karma-jasmine-ajax",
      "karma-jasmine"
    ],

    // base path that will be used to resolve all patterns (eg. files, exclude)
    basePath: "../../", // Frontend

    // frameworks to use
    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
    frameworks: ["jasmine-ajax", "jasmine"],

    // list of files / patterns to load in the browser
    files: CONFIG.files,

    // proxies
    proxies: CONFIG.proxies,

    // list of files / patterns to exclude
    exclude: [],

    // preprocess matching files before serving them to the browser
    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
    preprocessors: {},

    // test results reporter to use
    // possible values: "dots", "progress"
    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
    reporters: [
      "progress"
    ],

    // web server port
    port: 9876,

    // enable / disable colors in the output (reporters and logs)
    colors: true,

    // level of logging
    logLevel: config.LOG_INFO,

    // enable / disable watching file and executing tests whenever any file changes
    autoWatch: true,

    // immediately stop the current run and re-run the tests
    restartOnFileChange: true,

    // start these browsers
    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
    browsers: ["Chrome"], // "PuppeteerChrome", "Chrome"
    customLaunchers: {
      PuppeteerChrome: {
        base: "ChromeHeadless",
        flags: ["--no-sandbox", "--disable-setuid-sandbox", "--headless", "--disable-gpu"]
      }
    },
    browserNoActivityTimeout: 90000, // 1.5 mins

    browserDisconnectTolerance: 5,

    // Continuous Integration mode
    // if true, Karma captures browsers, runs the tests and exits
    singleRun: !IS_DEV_MODE, // single run for Production, instant for Development

    // Concurrency level
    // how many browser should be started simultaneous
    concurrency: Infinity
  }

- Content of `package.json` file

{
  "name": "my-karma-test",
  "version": "1.0.0",
  "scripts": {
    "test": "node ./node_modules/karma/bin/karma start karma.conf.js",
    "dev-test": "DEV=true node --inspect ./node_modules/karma/bin/karma start karma.conf.js"
  },
  "devDependencies": {
    "chalk": "^2.4.1",
    "jasmine-core": "^3.1.0",
    "karma": "^2.0.4",
    "karma-chrome-launcher": "^2.2.0",
    "karma-jasmine": "^1.1.2",
    "karma-jasmine-ajax": "^0.1.13",
    "puppeteer": "^1.5.0"
  }
}

Customizations: Not sure if that could be the reason behind the problem... Among the loaded files is my hooks.js, which delay karma.loaded until my ExtJS application is actually started. That works fine and I don't see how that can be a cause for a wrong Browser state. The content of my hooks.js is:

window.karmaLoadedFn = window.__karma__.loaded; // temporary buffer variable
window.immediatelyLoad = false;
window.__karma__.loaded = () => {
    immediatelyLoad = true;
};

console.log("Loading additional files in order to start an ExtJS application...");
window._isMyAppReady = window.setInterval(() => {
    // console.log(".");
    if (Ext && Ext.app && Ext.app.Application && Ext.app.Application.instance) {
        console.log("[KARMA HOOK] ExtJS application is successfuly started. Karma is marked as LOADED.");
        clearInterval(window._isMyAppReady);

        window.__karma__.loaded = window.karmaLoadedFn; // restore __karma__.loaded function
        if (window.immediatelyLoad) {
            window.__karma__.loaded(); // force karma load notification
        }
    }
}, 1000); // poll each second

Potential fix

I can easily fix the problem if I force the correct state of the browser in the onStart method. And it looks correct: once you in the Browser.onStart, you create a new instance of Browser.lastResult, and you set a fresh value of "Browser.lastResult.total", then you probably want to force the browser state to EXECUTING (as you've just started new test run).

But I don't get how it can happen that tests are started for the second time correctly, I can see the message "Executed 0 of 88 SUCCESS ( 0 secs / 0 secs )", then all my specs are actually executed, but the Browser.onResult just ignores the results because of the wrong state.

I didn't give up on that after my local fix, so I continue my investigation. Any idea or help with that issue is very much appreciated!

solo-md commented 6 years ago

Figured out that after the file_list_modified event is emitted, and the browser is forced to EXECUTE state, and before the onStart method, the state is actually changed to READY.

The most interesting is that it happend because meantime karma send it's 'complete' event. And the complete event is sent because some WebSocket gets an 42["stop"] event.

It is the case for my Win10 environment. Doesn't happen for macOS.

solo-md commented 6 years ago

OK, Looks like it's because of https://github.com/karma-runner/karma/issues/1640 Can be closed as duplicate

karma-runner / karma