karma-runner / karma-coverage

A Karma plugin. Generate code coverage.
MIT License
770 stars 247 forks source link

No coverage report generated for large project #434

Closed cklogs closed 2 years ago

cklogs commented 3 years ago

Hi there,

This library is now recommended for code coverage when using v11 of the angular-cli. We are currently using karma-coverage-istanbul-reporter because we have yet to get successful coverage reports with karma-coverage. Seeing, however, that there are deprecation warnings when using the istanbul reporter library with the v11 angular-cli, we would very much like to move off it.

Perhaps someone here can help discern what is wrong with our configuration or point us in the right direction.

I will include the karma.config, and test output debug logs for two different projects.

The first project (PROJECT-1) is a bare project that was initialized with ng new command. It was updated to Angular 11 using the ng update command from angular-cli. The point of including this project is to serve as a 'control' for the second project.

The second project (PROJECT-2) comes from a relatively large production codebase. In this codebase is roughly 5000 unit tests.

Important points

  1. The angular.json test configuration block, tsconfig.spec.ts, and karma.config for both PROJECT-1 and PROJECT-2 are exactly the same. I will post the shared karma.config below.
  2. v11 angular-cli still has memory management issues at build time, similar to those mentioned here. For build/run/test on PROJECT-2 we have to use the --max_old_space_size flag. Note that we are running with latest version of the angular-cli and node v12.x.

PROJECT-1 tests are run with: ng test. PROJECT-2 tests are run with: node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng test.

  1. When running tests on PROJECT-1, I get consistent coverage reports. I never see coverage reports for PROJECT-2.
  2. The only discernable difference between PROJECT-1 and PROJECT-2 is the --max_old_space_size configuration property mentioned about.

Below is the karma config file shared by the two projects and the debug logs that are output when running tests.

Karma config

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html

module.exports = function (config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine', '@angular-devkit/build-angular'],
    plugins: [
      require('karma-spec-reporter'),
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-coverage'),
      require('@angular-devkit/build-angular/plugins/karma')
    ],
    client: {
      clearContext: false // leave Jasmine Spec Runner output visible in browser
    },
    coverage: {
      dir: require('path').join(__dirname, '../coverage/karma-coverage'),
      reports: ['html', 'lcovonly', 'text-summary']
    },
    reporters: ['dots', 'spec', 'coverage'],
    specReporter: {
    maxLogLines: 5, // limit number of lines logged per test
    suppressErrorSummary: true, // do not print error summary
    suppressFailed: false, // do not print information about failed tests
    suppressPassed: true, // do not print information about passed tests
    suppressSkipped: true, // do not print information about skipped tests
    showSpecTiming: false, // print the time elapsed for each spec
    failFast: false // test would finish with error when a first fail occurs.
     },
    customLaunchers: {
    'CustomChromeHeadless': {
        base: 'ChromeHeadless',
        flags: [
            '--disable-gpu',
            '--disable-web-security',
            '--no-sandbox',
            '--remote-debugging-port=9222'
        ],
        debug: true
    }
    },
    browsers: ['CustomChromeHeadless'],
    browserDisconnectTolerance: 2,
    browserNoActivityTimeout: 300000, // five minutes
    browserDisconnectTimeout: 60000,
    captureTimeout: 300000,
    hostname: '127.0.0.1',
    port: 9876,
    colors: true,
    logLevel: config.LOG_DEBUG,
    autoWatch: false,
    singleRun: true,
    restartOnFileChange: false
  });
};

PROJECT-1-debug-output.txt PROJECT-2-debug-output.txt

In the PROJECT-2 output you will see only ~20 tests ran. This is because I used an fdescribe to run a single test file so as to reduce the noise a little. The same result--no coverage--happens with or without this change.

The point of failure seems to be here: image

Thanks in advance for your support.

alan-agius4 commented 3 years ago

@cklogs, karma-coverage needs to be configured using the coverageReporter property, which is missing from your configuration.

Can you please try to change the configuration like the below;

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html

module.exports = function (config) {
    config.set({
        basePath: '',
        frameworks: ['jasmine', '@angular-devkit/build-angular'],
        plugins: [
            require('karma-spec-reporter'),
            require('karma-jasmine'),
            require('karma-chrome-launcher'),
            require('karma-coverage'),
            require('@angular-devkit/build-angular/plugins/karma')
        ],
        client: {
            clearContext: false // leave Jasmine Spec Runner output visible in browser
        },
+       coverageReporter: {
+           dir: require('path').join(__dirname, './coverage/karma-coverage'),
+           subdir: '.',
+           reporters: [
+               { type: 'html' },
+               { type: 'text-summary' }
+            ]
+        },
-       coverage: {
-           dir: require('path').join(__dirname, '../coverage/karma-coverage'),
-           reports: ['html', 'lcovonly', 'text-summary']
-       },
        reporters: ['dots', 'spec', 'coverage'],
        specReporter: {
            maxLogLines: 5, // limit number of lines logged per test
            suppressErrorSummary: true, // do not print error summary
            suppressFailed: false, // do not print information about failed tests
            suppressPassed: true, // do not print information about passed tests
            suppressSkipped: true, // do not print information about skipped tests
            showSpecTiming: false, // print the time elapsed for each spec
            failFast: false // test would finish with error when a first fail occurs.
        },
        customLaunchers: {
            'CustomChromeHeadless': {
                base: 'ChromeHeadless',
                flags: [
                    '--disable-gpu',
                    '--disable-web-security',
                    '--no-sandbox',
                    '--remote-debugging-port=9222'
                ],
                debug: true
            }
        },
        browsers: ['CustomChromeHeadless'],
        browserDisconnectTolerance: 2,
        browserNoActivityTimeout: 300000, // five minutes
        browserDisconnectTimeout: 60000,
        captureTimeout: 300000,
        hostname: '127.0.0.1',
        port: 9876,
        colors: true,
        logLevel: config.LOG_DEBUG,
        autoWatch: false,
        singleRun: true,
        restartOnFileChange: false
    });
};

More information about karma-coverage configuration can be found here: https://github.com/karma-runner/karma-coverage/blob/master/docs/configuration.md

cklogs commented 3 years ago

@alan-agius4

I tried exactly as is suggested above and coverage reports are still not being generated. Perhaps there is something more I'm missing?

alan-agius4 commented 3 years ago

Without a reproduction it's hard to tell what's happening. You can try to debug https://github.com/karma-runner/karma-coverage/blob/1d34a56ba612ee3970c99c2e2a4822a50870384c/lib/reporter.js#L224 to see what's happening.

cklogs commented 3 years ago

I'm still trying to demystify what is going on but I have observed the following: For these particular methods:

https://github.com/karma-runner/karma-coverage/blob/1d34a56ba612ee3970c99c2e2a4822a50870384c/lib/reporter.js#L185-L201

The browsers object on line 185 is defined as:

BrowserCollection {
  browsers: [],
  emitter: EventEmitter {
    _events: [Object: null prototype] {},
    _eventsCount: 0,
    _maxListeners: undefined,
    [Symbol(kCapture)]: false
  }
}

This means that if (browsers) { executes.

However, this.onBrowserStart is never called during the entire execution of the tests.

Once the tests have all ran, this function is called:

https://github.com/karma-runner/karma-coverage/blob/1d34a56ba612ee3970c99c2e2a4822a50870384c/lib/reporter.js#L203-L210

The parameter browser is defined as:

Browser {
  id: '35804940',
  fullName: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/89.0.4389.90 Safari/537.36',
  name: 'Chrome Headless 89.0.4389.90 (Windows 10)',
  lastResult: BrowserResult {
    startTime: 1617310253880,
    total: 0,
    success: 6032,
    failed: 0,
    skipped: 0,
    totalTime: 628830,
    netTime: 437749,
    error: false,
    disconnected: false
  },
  disconnectsCount: 0,
  activeSockets: [
    Socket {
      _events: [Object: null prototype],
      _eventsCount: 8,
      _maxListeners: undefined,
      nsp: [Namespace],
      client: [Client],
      acks: Map {},
      fns: [],
      flags: {},
      _rooms: Set {},
      server: [Server],
      adapter: [Adapter],
      id: 'eEVk_hCsxP6b3WzbAAAD',
      connected: true,
      disconnected: false,
      handshake: [Object],
      [Symbol(kCapture)]: false
    }
  ],
  noActivityTimeout: 300000,
  singleRun: true,
  clientConfig: {
    args: [],
    useIframe: true,
    runInParent: false,
    captureConsole: true,
    clearContext: true,
    originalArgs: []
  },
  collection: BrowserCollection {
    browsers: [ [Circular] ],
    emitter: Server {
      _events: [Object: null prototype],
      _eventsCount: 17,
      _maxListeners: undefined,
      log: [Logger],
      loadErrors: [],
      _injector: [Injector],
      _boundServer: [Server],
      _fileList: [FileList],
      [Symbol(kCapture)]: false
    }
  },
  emitter: Server {
    _events: [Object: null prototype] {
      load_error: [Function],
      exit: [Array],
      run_complete: [Array],
      browser_complete: [Array],
      file_list_modified: [Array],
      run_start: [Array],
      browser_start: [Array],
      browser_error: [Array],
      browser_log: [Array],
      spec_complete: [Array],
      spec_failure: [Function],
      browsers_change: [Function],
      browser_register: [Function],
      stop: [Function],
      browser_restart_failure: [Function],
      browser_complete_with_no_more_retries: [Function],
      browser_process_failure: [Function]
    },
    _eventsCount: 17,
    _maxListeners: undefined,
    log: Logger {
      category: 'karma-server',
      context: {},
      parseCallStack: [Function: defaultParseCallStack]
    },
    loadErrors: [],
    _injector: Injector {
      _providers: [Object: null prototype],
      _instances: [Object: null prototype],
      get: [Function: get],
      invoke: [Function: invoke],
      instantiate: [Function: instantiate],
      createChild: [Function: createChild]
    },
    _boundServer: Server {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      _connections: 0,
      _handle: [TCP],
      _usingWorkers: false,
      _workers: [],
      _unref: false,
      allowHalfOpen: false,
      pauseOnConnect: false,
      _connectionKey: '4:0.0.0.0:9876',
      [Symbol(kCapture)]: false,
      [Symbol(asyncId)]: 13
    },
    _fileList: FileList {
      _patterns: [Array],
      _excludes: [Array],
      _emitter: [Circular],
      _preprocess: [AsyncFunction: preprocess],
      buckets: [Map],
      _refreshing: [Promise],
      _emitModified: [Function]
    },
    [Symbol(kCapture)]: false
  },
  socket: Socket {
    _events: [Object: null prototype] {
      start: [Function],
      info: [Function],
      karma_error: [Function],
      result: [Function],
      complete: [Function],
      error: [Function],
      register: [Function],
      disconnect: [Function]
    },
    _eventsCount: 8,
    _maxListeners: undefined,
    nsp: Namespace {
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      sockets: [Map],
      _fns: [],
      _rooms: Set {},
      _flags: {},
      _ids: 0,
      server: [Server],
      name: '/',
      adapter: [Adapter],
      [Symbol(kCapture)]: false
    },
    client: Client {
      sockets: Map {},
      nsps: Map {},
      server: [Server],
      conn: [Socket],
      encoder: Encoder {},
      decoder: [Decoder],
      id: '2DlQ3r3gH89vkFHCAAAA',
      onclose: [Function: bound onclose],
      ondata: [Function: bound ondata],
      onerror: [Function: bound onerror],
      ondecoded: [Function: bound ondecoded],
      connectTimeout: undefined
    },
    acks: Map {},
    fns: [],
    flags: {},
    _rooms: Set {},
    server: Server {
      _events: [Object: null prototype] {},
      _eventsCount: 0,
      _maxListeners: undefined,
      _nsps: [Map],
      parentNsps: Map {},
      _path: '/socket.io',
      clientPathRegex: /^\/socket\.io\/socket\.io(\.min|\.msgpack\.min)?\.js(\.map)?$/,
      _connectTimeout: 45000,
      _serveClient: true,
      _parser: [Object],
      encoder: Encoder {},
      _adapter: [class Adapter extends EventEmitter],
      sockets: [Namespace],
      opts: [Object],
      eio: [Server],
      httpServer: [Server],
      engine: [Server],
      [Symbol(kCapture)]: false
    },
    adapter: Adapter {
      _events: [Object: null prototype] {},
      _eventsCount: 0,
      _maxListeners: undefined,
      nsp: [Namespace],
      rooms: [Map],
      sids: [Map],
      encoder: Encoder {},
      [Symbol(kCapture)]: false
    },
    id: 'tpLs3SFHdGeiHnz_AAAB',
    connected: false,
    disconnected: true,
    handshake: {
      headers: [Object],
      time: 'Thu Apr 01 2021 13:50:53 GMT-0700 (Pacific Daylight Time)',
      address: '127.0.0.1',
      xdomain: false,
      secure: false,
      issued: 1617310253837,
      url: '/socket.io/?EIO=4&transport=polling&t=NYFD0BQ',
      query: [Object: null prototype],
      auth: {}
    },
    [Symbol(kCapture)]: false
  },
  timer: {
    setTimeout: [Function: setTimeout],
    clearTimeout: [Function: clearTimeout]
  },
  disconnectDelay: 60000,
  log: Logger {
    category: 'Chrome Headless 89.0.4389.90 (Windows 10)',
    context: {},
    parseCallStack: [Function: defaultParseCallStack]
  },
  noActivityTimeoutId: Timeout {
    _idleTimeout: 300000,
    _idlePrev: [TimersList],
    _idleNext: [TimersList],
    _idleStart: 983207,
    _onTimeout: [Function],
    _timerArgs: undefined,
    _repeat: null,
    _destroyed: false,
    [Symbol(refed)]: true,
    [Symbol(asyncId)]: 1889860,
    [Symbol(triggerId)]: 1889856
  },
  pendingDisconnect: Timeout {
    _idleTimeout: -1,
    _idlePrev: null,
    _idleNext: null,
    _idleStart: 385875,
    _onTimeout: null,
    _timerArgs: undefined,
    _repeat: null,
    _destroyed: true,
    [Symbol(refed)]: true,
    [Symbol(asyncId)]: 1809518,
    [Symbol(triggerId)]: 1809514
  },
  state: 'CONNECTED'
}

coverageMaps are defined as:

[Object: null prototype] {}

The result is a large object containing coverage results for each file. One such example:

   'C:\\app\\shared\\helpers\\url-serializer.ts': {
      path: 'C:\\app\\shared\\helpers\\url-serializer.ts',
      statementMap: [Object],
      fnMap: [Object],
      branchMap: [Object],
      s: [Object],
      f: [Object],
      b: [Object],
      inputSourceMap: [Object],
      _coverageSchema: '1a1c01bbd47fc00a2c39e90264f33305004495a9',
      hash: 'b12799a1103f348e9c0f91bb2cb5bbf81c79d909'
    }

Is there anything more you would like to see?

cklogs commented 3 years ago

@alan-agius4 When you have a chance, could you please review my updates? Advance thanks!

alan-agius4 commented 3 years ago

I'll leave the triaging to someone from the karma-coverage team.

cklogs commented 3 years ago

To the maintainers of this repo:

I have a small suspicion that,

  1. the sheer size of our project and,
  2. the associated long compile time while running tests

have something to do with coverage failing to be generated. But this is only a hunch.

I cannot share or anonymize our code base, but if there is a large open repository that you would suggest I try and reproduce this issue from, I will happily try.

Aside from that, if you have any insights on the above issue, please let me know here or contact me directly.

Thanks again.

WebMex commented 3 years ago

We also have a big project with 3000+ unit tests. And I can see coverage reports generated only on powerful machines. On our ci server and on slow machines it's not generated.

cklogs commented 3 years ago

@WebMex Thank you for the additional data point!

Do you plan to continue using karma-coverage, or have you sought out a different library? We have not been able to move onto karma-coverage (from karma-coverage-instanbul-reporter) because of the above issue, but we are open to other coverage generators.

WebMex commented 3 years ago

@cklogs I have rolled back to karma-coverage-instanbul-reporter so far and waiting for the fix. Despite the warning message from angular it is still possible to use it.

cklogs commented 3 years ago

@alan-agius4

So far I have had little luck getting feedback from any of the maintainers of this library. Given karma-coverage is now the defacto standard coverage library for Angular, it may be necessary to raise this up to the greater Angular team.

Please advise on how to most effectively do this.

Thanks in advance for your help!

cklogs commented 3 years ago

@WebMex

Thanks to your tip, I tested karma-coverage on a faster box (Ryzen 7 PRO / 32 GB RAM), and I am getting consistent coverage results there. Still need this fix for our deployment servers to generate coverage, but I'm hoping these multiple data points give the maintainers a lead on the issue.

alan-agius4 commented 3 years ago

This would be a hard one to track down without a reproduction. I tried it on a seed project and generated a large number of tests but I didn't manage to replicate.

Unfortunately, I am not too familiar with this plugin to get to the bottom of this without being able to reproduce the issue.

arobinson commented 3 years ago

I am seeing the same issue. If I run the test on my mac locally, I'm seeing the lcov.info file always get generated. However, running the test with even a greater --max_old_space_size size specified and run by Jenkins on a linux box intermittently does not generate the lcov.info file. There are no reports of exceptions or anything else I can see that explain why the report is not generated.

When the coverage isn't generated I see something like:

[2021-06-29T21:37:31.258Z] TOTAL: 6266 SUCCESS
[2021-06-29T21:37:31.258Z] 
[2021-06-29T21:37:31.258Z] Finished in 13 mins 7.178 secs / 9 mins 58.633 secs @ 21:37:31 GMT+0000 (Coordinated Universal Time)
[2021-06-29T21:37:31.258Z] 
[2021-06-29T21:37:31.258Z] SUMMARY:
[2021-06-29T21:37:31.258Z] ✔ 6266 tests completed
[2021-06-29T21:37:31.258Z] ℹ 28 tests skipped

When I run it locally and the file is generated I get:

TOTAL: 6266 SUCCESS

Finished in 14 mins 0.813 secs / 10 mins 58.549 secs @ 15:34:47 GMT-0600 (Mountain Daylight Time)

SUMMARY:
✔ 6266 tests completed
ℹ 28 tests skipped
29 06 2021 15:35:04.211:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.

Oddly, it seems the Chrome was not killed is usually reported when the lcov.info file is generated, but if that is not reported, the coverage file was not generated. So it almost seems like the node process is exiting before the karma-coverage plugin is done generating the report.

Could the problem be that the process can exit before the plugin/reporter has completed writing the file?

cklogs commented 3 years ago

@alan-agius4 Perhaps this project might be a good place to run a smoke test of this issue: https://github.com/angular/angular-cli-stress-test. I was able to get it roughly running just by dropping the files into a newly spun up Angular 12 project. I could not get coverage reports generating on the codebase. You will likely be able to reproduce the issue so long as your machine is not too powerful.

nukithelegend commented 3 years ago

Please can this be done? We have the same issue. We have about 13000 unit tests. Sometimes the coverage report gets generated, however it mostly fails. We run them on a Linux docker image trion/ng-cli-karma.

It's upsetting to be "punished" for having too many tests.

nukithelegend commented 3 years ago

I have done a previous investigation on this, and I found that it was related to the speed/number of tests. It failed in Istanbul, I believe it was when merging the coverage summaries that the result was just suddenly empty and then no coverage were created

arobinson commented 3 years ago

Unfortunately this started happening ~75% of the time for us, but I could not reproduce it locally. Only solution I could find was to uninstall and go back to an unmaintained, but much more reliable solution for now. This is a blocker, we need it to be 100% reliable.

arobinson commented 3 years ago

This looks to be an issue with Karma: https://github.com/karma-runner/karma/issues/945

Unfortunately it seems that Karma is a dead project. That issue was opened in 2014 and was closed since no one worked on the issue

jginsburgn commented 3 years ago

It is absolutely not dead! We work with limited resources, so we cannot address every issue immediately and sadly that one got closed a long time ago; I will be adding it to my working queue. Also, your collaboration is welcome.

lzilioli-prism commented 3 years ago

I am experiencing the same issue after addressing the following message from our test run:

'karma-coverage-istanbul-reporter' usage has been deprecated since version 11.
            Please install 'karma-coverage' and update 'karma.conf.js.' For more info, see https://github.com/karma-runner/karma-coverage/blob/master/README.md

I am going to revert the change in my project, and follow this issue, in hopes it gets resolved soon.

Some additional observations: I have seen the coverage report generated one time successfully, and if I scope the files in my test runner to a smaller subset of .spec.ts files, it is generated every time. This clearly seems to be a memory or cpu issue. Whats troubling to me is that the process exits as though it successfully generated the report!

lzilioli-prism commented 3 years ago

Update: Reverting back to karma-coverage-istanbul-reporter 100% fixed the issue, and our coverage reports are being reliably generated again, despite the deprecation warning. I can confirm there are issues with code coverage reports and:

Angular:

$ ng --version

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/

Angular CLI: 12.2.1
Node: 14.15.5
Package Manager: npm 6.14.11
OS: darwin x64

Angular: 12.2.1
... animations, cdk, cli, common, compiler, compiler-cli, core
... forms, language-service, material, material-moment-adapter
... platform-browser, platform-browser-dynamic, platform-server
... router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1202.1
@angular-devkit/build-angular   12.2.1
@angular-devkit/core            12.2.1
@angular-devkit/schematics      12.2.1
@angular/flex-layout            12.0.0-beta.34
@schematics/angular             12.2.1
rxjs                            6.6.7
typescript                      4.3.5
webpack                         5.50.0

karma stuffs from package.json:

$ cat package.json | grep karma
        "karma-spec-reporter": "^0.0.32",
        "karma": "~6.3.4",
        "karma-chai": "^0.1.0",
        "karma-chrome-launcher": "~3.1.0",
        "karma-cli": "^2.0.0",
        "karma-coverage": "~2.0.3",
        "karma-jasmine": "~4.0.0",
        "karma-jasmine-html-reporter": "^1.5.0",
        "karma-junit-reporter": "^2.0.1",
        "karma-phantomjs-launcher": "^1.0.4",
        "karma-typescript": "^5.5.1",

I hope this helps

rrauenza commented 3 years ago

I think I am seeing this problem as well. Intermittently, the coverage directory isn't created. I don't know that my project is large or not... not sure of the baseline!

$ ./node_modules/.bin/cloc --include-lang=TypeScript src/app/ vendored/
     381 text files.
     380 unique files.                                          
     148 files ignored.

github.com/AlDanial/cloc v 1.90  T=1.48 s (165.3 files/s, 117159.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
TypeScript                     245           2320           1451         169847
-------------------------------------------------------------------------------
SUM:                           245           2320           1451         169847
-------------------------------------------------------------------------------

An example of failure:

    ✓ should submit form
    ✓ should submit form API data
    ✓ should check permission for edit
    ✓ should edit & delete API key
TOTAL: 182 SUCCESS
TOTAL: 182 SUCCESS
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 0 (skipped 1) SUCCESS (59.589 secs / 17.681 secs)
TOTAL: 182 SUCCESS
 ____  _   _  ____ ____ _____ ____ ____  
/ ___|| | | |/ ___/ ___| ____/ ___/ ___| 
\___ \| | | | |  | |   |  _| \___ \___ \ 
 ___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/ 

$ ls -la coverage/
ls: cannot access coverage/: No such file or directory
Cleaning up file based variables

An example of success:

Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 183 (skipped 1) SUCCESS (17.465 secs / 15.649 secs)
TOTAL: 182 SUCCESS
TOTAL: 182 SUCCESS
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 183 (skipped 1) SUCCESS (17.465 secs / 15.649 secs)
TOTAL: 182 SUCCESS
08 10 2021 11:15:39.085:WARN [launcher]: ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
 ____  _   _  ____ ____ _____ ____ ____  
/ ___|| | | |/ ___/ ___| ____/ ___/ ___| 
\___ \| | | | |  | |   |  _| \___ \___ \ 
 ___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/ 

$ ls -la coverage/
total 832
drwxrwxr-x 4 gitlab-runner gitlab-runner    152 Oct  8 11:15 .
drwxrwxr-x 8 gitlab-runner gitlab-runner   4096 Oct  8 11:15 ..
-rw-rw-r-- 1 gitlab-runner gitlab-runner 658117 Oct  8 11:15 cobertura.txt
drwxrwxr-x 3 gitlab-runner gitlab-runner    182 Oct  8 11:15 report-html
drwxrwxr-x 3 gitlab-runner gitlab-runner     42 Oct  8 11:15 report-lcov
-rw-rw-r-- 1 gitlab-runner gitlab-runner 160020 Oct  8 11:15 report-lcovonly.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner    697 Oct  8 11:15 teamcity.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner    302 Oct  8 11:15 text-summary.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner  14499 Oct  8 11:15 text.txt

What is kind of interesting is that I get the SIGKILL in the case in which it works.... but maybe that's just karma killing the browser as it writes out the reports.

I also had no problems when using Istanbul previously.

Due to the warning, I'm just going to arbitrarily try setting the timeout higher in karma.conf.js and see what happens... I can't justify why it might help, but I'd like to see the warning go away anyway:

processKillTimeout: 60 * 1000

update: it didn't help. I still get intermittent failures to create the coverage output. update 2: reducing the number of kinds of reports to just text/text-summary might have helped. Waiting for more data. update 3: seems to still happen. Maybe when cicd machine is under higher load. Will need to go back to istanbul version more than likely.

ozkoidi commented 3 years ago

I have exactly the same issue. The coverage reports are generated only when the following messasge is logged: ChromeHeadless was not killed in 2000 ms, sending SIGKILL.

Is there a way to force karma to always send this? Or any other updates on this issue?

devoto13 commented 2 years ago

From a quick look it seems that the problem is that onRunComplete returns a Promise, but Karma does not wait for for it. As a result Karma shutdown will race with the writing of the coverage report.

Unfortunately, without a reproduction it is hard to confirm or reject the theory. I wonder if somebody having the problem can run tests using this branch and see if it solves the problem? The only change there is that Karma will not call process.exit once done, thus allowing all async activities to complete.

npm i devoto13/karma#test-coverage

UPD Looks like that promise is actually awaited in the onExit callback. Another place, which is not synchronized is this one 🤔 I think it still will be useful to perform the test, so we can confirm that the issue is caused by the race and focus on locating and fixing this race.

arobinson commented 2 years ago

I wonder if somebody having the problem can run tests using this branch and see if it solves the problem?

Trying now. I'll do a few runs on this and report back to see how it does with our code. Thanks

arobinson commented 2 years ago

First run worked. Our CI server is busy today and the tests can take a while, so I'll run a few more runs on the branch and see if it is working 100% of the time across several runs.

ozkoidi commented 2 years ago

I tried the branch that @devoto13 posted and it does seem to fix the problem 👍 Out of 20 executions the coverage report was generated in all of them.

devoto13 commented 2 years ago

Thanks for the confirmation @ozkoidi! So it seems that indeed the problem is the race between the Karma shutdown and the writing of the coverage. So far this looks like the main offender. I'll try to put a PR to properly wait for it later today.

arobinson commented 2 years ago

I did not have any reproductions of the issue during my checks on the branch. We are going to try to use it for all our code until it is released and I'll report back if anyone fails to have coverage generated. Thank you!

devoto13 commented 2 years ago

@arobinson The branch was only intended for testing the race, I think it will always exit with 0 exit code (even when tests have failed).

devoto13 commented 2 years ago

Haven't found any more races, so hopefully, https://github.com/karma-runner/karma-coverage/pull/463 is the final fix. You can also give it a try on your codebase by installing it from the branch using the below command:

$ npm i devoto13/karma-coverage#fix-race
arobinson commented 2 years ago

I think it will always exit with 0 exit code

FYI, I got an exit code of 1 with a test failure even with that branch

arobinson commented 2 years ago

Using the fix-race branch for https://github.com/karma-runner/karma-coverage/pull/463, I tried it on our software and the build failed:

Test report file /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/junit/TESTS-Chrome_Headless_92.0.4515.159_(Linux_x86_64).xml was length 0

The tests started fine:

[2022-01-20T16:15:59.383Z] - Generating browser application bundles (phase: setup)...
[2022-01-20T16:16:01.932Z] Test results are being saved to: /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results
[2022-01-20T16:16:01.932Z] 20 01 2022 16:16:01.915:INFO [karma-server]: Writing browser console to file: /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/console.log
[2022-01-20T16:16:01.932Z] 
[2022-01-20T16:16:01.932Z] START:
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.938:INFO [karma-server]: Karma v6.3.11 server started at http://localhost:9876/
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.939:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.949:INFO [launcher]: Starting browser Chrome
[2022-01-20T16:18:08.605Z] 20 01 2022 16:18:05.217:WARN [launcher]: Chrome has not captured in 120000 ms, killing.
[2022-01-20T16:18:08.605Z] 20 01 2022 16:18:07.315:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
[2022-01-20T16:18:20.800Z] 20 01 2022 16:18:19.493:INFO [launcher]: Trying to start Chrome again (1/2).
[2022-01-20T16:19:42.190Z] ✔ Browser application bundle generation complete.
[2022-01-20T16:19:42.190Z] 20 01 2022 16:19:33.627:INFO [Chrome Headless 92.0.4515.159 (Linux x86_64)]: Connected on socket uvg6glKjq65iOVWJAAAB with id 95294800
[2022-01-20T16:20:04.076Z]   AppComponent Test Suite
[2022-01-20T16:20:04.076Z]     ✔ Initial Rendering

But at the end, the tests failed badly:

[2022-01-20T16:31:58.196Z] ✔ 7564 tests completed
[2022-01-20T16:31:58.196Z] ℹ 35 tests skipped
[2022-01-20T16:32:13.033Z] 20 01 2022 16:32:12.590:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.792:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.792:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z]   errno: -17,
[2022-01-20T16:32:13.968Z]   code: 'EEXIST',
[2022-01-20T16:32:13.968Z]   syscall: 'mkdir',
[2022-01-20T16:32:13.968Z]   path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.794:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.794:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z]   errno: -17,
[2022-01-20T16:32:13.968Z]   code: 'EEXIST',
[2022-01-20T16:32:13.968Z]   syscall: 'mkdir',
[2022-01-20T16:32:13.968Z]   path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z]   errno: -17,
[2022-01-20T16:32:13.968Z]   code: 'EEXIST',
[2022-01-20T16:32:13.968Z]   syscall: 'mkdir',
[2022-01-20T16:32:13.968Z]   path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z]   errno: -17,
[2022-01-20T16:32:13.968Z]   code: 'EEXIST',
[2022-01-20T16:32:13.968Z]   syscall: 'mkdir',
[2022-01-20T16:32:13.968Z]   path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }

That EEXIST error was repeated 34,494 times in the job log before Jenkins stopped recording the output. Looks like it got caught in an infinite loop

devoto13 commented 2 years ago

Thanks for the feedback @arobinson. Looks like there is still a race between a check for a directory existence and creation in the helper, which is executed concurrently for all reporters. I'll look into fixing it.

Interesting that it got into an infinite loop though 🤔

devoto13 commented 2 years ago

Here is another attempt to fix this:

$ npm i devoto13/karma-coverage#fix-race
$ npm i devoto13/karma#fix-helper

which should address the infinite loop and the race when creating the same directory concurrently. I've tried it on the fresh Angular project and it seems to work now.

cklogs commented 2 years ago

@alan-agius4 I've noticed since upgrading our project to Angular 13 that, when running ng test, the karma-coverage-instanbul-reporter deprecation warning has gone away.

Do you know if this was this intentional? I combing through the v13 commits to angular-cli, I could not find where it was changed. I thought you might have some insight.

alan-agius4 commented 2 years ago

Yea, that’s intentional as we no longer support it.

that said, you can still use it, if you provide the reporter configuration yourself.

See: https://github.com/angular/angular-cli/pull/21493

arobinson commented 2 years ago

Testing the latest patches, I did get code coverage numbers without any issues. ~I'll try a couple of more times to make sure~

Did 3 runs, all collected coverage correctly

devoto13 commented 2 years ago

@jginsburgn Looks like my changes resolve the coverage flakiness. Please take a look at the corresponding PRs (one and two) when you have some time.

jginsburgn commented 2 years ago

@jginsburgn Looks like my changes resolve the coverage flakiness. Please take a look at the corresponding PRs (one and two) when you have some time.

Checking them now. Sorry about my delay.

karmarunnerbot commented 2 years ago

:tada: This issue has been resolved in version 2.1.1 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

rrauenza commented 2 years ago

I upgraded and I believe I'm still not getting a coverage directory in cicd. Locally, yes, but in cicd, no. The main difference being the load being higher on the cicd machine from other jobs running in parallel.

$ grep karma-coverage yarn.lock 
karma-coverage-istanbul-reporter@^3.0.3:
  resolved "http://build-artifactory.eng.vmware.com/artifactory/api/npm/npm/karma-coverage-istanbul-reporter/-/karma-coverage-istanbul-reporter-3.0.3.tgz#f3b5303553aadc8e681d40d360dfdc19bc7e9fe9"
karma-coverage@^2.1.1:
  resolved "http://build-artifactory.eng.vmware.com/artifactory/api/npm/npm/karma-coverage/-/karma-coverage-2.1.1.tgz#7434f8736841e08de3e6c302632e8d239634f388"

(I'll also try without the istanbul one also in node_modules)

$ more karma.conf.js 

const path = require("path")
const fs = require("fs")

function get_backend_url() {
  const config_file = path.resolve(process.cwd(), ".karma-backend-config.json")
  try {
    const data = fs.readFileSync(config_file)
    const config = JSON.parse(data)
    return config.LIVE_SERVER_URL
  } catch (error) {
    return "http://localhost:12345"
  }
}

const backend_url = get_backend_url()

module.exports = function (config) {
  config.set({
    failOnSkippedTests: false,
    basePath: "",
    frameworks: ["jasmine", "@angular-devkit/build-angular"],
    processKillTimeout: 60 * 1000,
    plugins: [
      require("karma-jasmine"),
      require("karma-chrome-launcher"),
      require("karma-coverage"),
      require("@angular-devkit/build-angular/plugins/karma"),
      require("karma-spec-reporter"),
    ],
    logLevel: config.LOG_INFO,
    client: {
      clearContext: false, // leave Jasmine Spec Runner output visible in browser
      jasmine: {
        random: false,
        timeoutInterval: 300000,
      },

    },

    // doesn't always work:  https://github.com/karma-runner/karma-coverage/issues/434
    coverageReporter: {
      dir: path.join(__dirname, "coverage"),
      reporters: [
        { type: "html", subdir: "report-html" },
        { type: "lcov", subdir: "report-lcov" },
        { type: "text-summary", subdir: ".", file: "text-summary.txt" },
      ],
      fixWebpackSourcePaths: true,
    },

    // https://stackoverflow.com/questions/54744584/karma-disconnected-because-no-message-in-10000-ms
    captureTimeout: 300000,
    browserDisconnectTolerance: 3,
    browserDisconnectTimeout: 300000,
    browserNoActivityTimeout: 300000,

    reporters: [
      "progress",
      "spec",
      "coverage",
    ],
    specReporter: {
      suppressSkipped: false,
    },
    hostname: "0.0.0.0",
    port: 9876,
    colors: true,
    autoWatch: true,
    browsers: ["ChromeHeadless"],
    singleRun: false,
    restartOnFileChange: true, 
    proxies: {
      "/api/": backend_url + "/api/",
      "/ajax/": backend_url + "/ajax/",
      "/saml2/": backend_url + "/saml2/",
    },
  })
}
devoto13 commented 2 years ago

@rrauenza Are you also using karma 6.3.15 which contains the second part of the fix?

If so I wonder if this can be related to the issue reported in https://github.com/karma-runner/karma/issues/3745, where onBrowserStart is not called. Would you be willing to add some logs to the on* hooks in https://github.com/karma-runner/karma-coverage/blob/9e8492c78dd92a5b7d906d82001104c42d4f9a7e/lib/reporter.js#L189 and run the same code locally and in your CI to pinpoint what exactly is not executed/executed differently? I think onBrowserStart should be a good place to start. With this information, I may be able to pinpoint the issue and make the fix.

The thing with onBrowserStart is that it is sent over the socket.io connection by the browser and I suspect that socket.io may lose messages while upgrading to the WebSocket connection when the host/browser is under heavy load. There were several issues reported that seem related, but it is tricky to reproduce and confirm from our side.

@arobinson I wonder if you had time to upgrade karma and karma-coverage in your project and can confirm whether the flaky coverage issues was resolved or it is still flaky for you?

rrauenza commented 2 years ago

@rrauenza Are you also using karma 6.3.15 which contains the second part of the fix?

No! I have 6.3.4 according to my yarn.lock. I'll upgrade. Does karma-coverage not require the newer version in its dependency requirements?

devoto13 commented 2 years ago

No, karma plugins are not used to have peerDependencies, that's something I would like to better address in the future.

ozkoidi commented 2 years ago

Unfortunately the code coverage is often not generated for me when the unit tests run on a Azure Devops pipeline (always works on local). I'm using the latest packages:

Here is a comparison of the end of the logs in case it helps. Since the source code is the same on both executions, I wonder if the difference in the order of the logged actions is the cause of the issue.

With test coverage generated 👍

image

Without test coverage: 👎

image

devoto13 commented 2 years ago

@ozkoidi From your logs, you seem to be using the https://github.com/joeljeske/karma-parallel plugin which may be the reason. Can you please try to run without it and see if you still get the same problem?

I'm not familiar with that plugin and will try to look at it when I have time to see if it does something suspicious.

ozkoidi commented 2 years ago

@devoto13, thanks for your suggestion. I removed the karma-parallel package and the result is still the same, so no need for you to look into that package.

In case it helps the specs of the Microsoft hosted agents running the tests are: 2 core CPU, 7 GB of RAM, and 14 GB of SSD disk space. I think those VM should be enough to run the test properly but I don't know what else could make the test not generate coverage reports randomly. Updated logs:

With test coverage generated 👍

image

Without test coverage 👎

image

devoto13 commented 2 years ago

@ozkoidi Thank you for the quick feedback. Can you please install karma and karma-coverage from these two branches and make more sample runs on your CI?

$ npm i devoto13/karma#test-coverage
$ npm i devoto13/karma-coverate#test-coverage

The first one removes the call to the process.exit on Karma shut down, so if the problem is resolved by the change, it means that we still have a race somewhere in the coverage generation routine. Per https://github.com/karma-runner/karma-coverage/issues/434#issuecomment-1013479221. Here is the code.

The second one is based on https://github.com/karma-runner/karma-coverage/issues/434#issuecomment-1033808800. It adds logs to all karma-coverage hooks so that we can compare good and bad runs and see if we can get some clue about what is different. Here is the code.