Coverage configuration with fast run and debug

segrey commented 9 years ago

As expected coverage slows down test run and makes debugging impossible. The workaround is to maintain two configs - the first one with coverage configured, and the second without coverage. To improve the situation, there is a feature that allows external integrations (e.g. IDE) to turn off coverage preprocessor by removing coverage reporter: https://github.com/karma-runner/karma-coverage/blob/v0.5.1/lib/preprocessor.js#L54 This feature has been using by IntelliJ integration until a configuration that need coverage preprocessor for test run has emerged: https://github.com/karma-runner/karma-coverage/blob/v0.5.1/examples/coffee/karma.conf.coffee

Now IntelliJ uses a new trick to perform "Run", "Debug" and "Run with Coverage" actions with single karma.conf.js. However, it expects one thing from users - they should remove coverage reporter if coverage preprocessor is not needed for test run (still common use case, IMO):

"Run" and "Debug" actions will work as coverage reporter is not set in config (done by users)
"Run with Coverage" will work because coverage reporter is added on-the-fly by IntelliJ integration (https://github.com/karma-runner/karma-intellij/blob/2f63c75ea451d25fb17346abf8813e611d5e94f7/lib/intellijCoverageReporter.js#L19).

The downsides of the current approach:

Undiscoverable for users
Coverage won't be gathered if tests are run outside of IntelliJ, e.g. in Terminal

It'd be appreciated if it would be possible to improve the situation. A possible solution could be changing current coffeescript recommended configuration by making it possible to run tests without required coverage preprocessor. Also that would speed up tests. Thanks.

segrey commented 9 years ago

Related WebStorm issue: https://youtrack.jetbrains.com/issue/WEB-17537

dignifiedquire commented 9 years ago

I'm sorry, maybe I'm being thick but what exactly would you like to see ideally in terms of features from karma/karma-coverage?

segrey commented 9 years ago

No problem. Generally, I would like to see a clear way that would allow to run(fast)/debug/run-with-coverage using single configuration file. Previously, there was some way, but as it turned out that way does not play well with recommended coffee configuration. For example, the fix could be in changing this recommended configuration (not sure it's the best solution).

segrey commented 8 years ago

@Dignifiedquire I've changed slightly how IntelliJ integration disables coverage to have faster test runs/debug capabilities. Could you please take a look at https://github.com/karma-runner/karma-intellij/commit/7b5ab8fcac7b10530836149b453f2a78c183d543 and let me your opinion on this. Personally, I think that checking for instrument and instruments keys is a bit odd and fragile, but it allows to fix all known cases. Probably, it can be improved somehow.

New logic:

  if (canCoverageBeDisabledSafely(config.coverageReporter)) {
    var ind = reporters.indexOf(karmaCoverageReporterName);
    if (ind >= 0) {
      reporters.splice(ind, 1);
      console.log('IntelliJ integration disabled coverage for faster run and debug capabilities');
    }
  }

/**
 * @param {Object} coverageReporter
 * @returns {boolean} true if tests can be successfully run without coverage reporter and preprocessor
 */
function canCoverageBeDisabledSafely(coverageReporter) {
  return coverageReporter == null || (
      !Object.prototype.hasOwnProperty.call(coverageReporter, 'instrumenter') &&
      !Object.prototype.hasOwnProperty.call(coverageReporter, 'instrumenters')
    );
}

This logic tries to preserve coverage reporter for configurations, like in https://github.com/karma-runner/karma-coverage/blob/v0.5.3/examples/coffee/karma.conf.coffee, where tests do not run without coverage. But for majority of karma test configurations, where tests can run successfully without coverage, this logic will disable coverage resulting in faster test runs and having debug capability.

karma-runner / karma-coverage

Coverage configuration with fast run and debug #183