Coverage for untested files?

[ZebraFlesh]

How do I use the coverage for untested files feature introduced in the 4.1.0 release? I reviewed the PR and it looks like I have to specify an onCover function, however, I'm currently specifying my istanbul config via a .babelrc file:

{
  "env": {
    "test": {
      "plugins": [
        [ "istanbul", {
          "exclude": [
            "tests/index.js",
            "tests/testutils/**/*.js",
            "**/*.spec.js"
          ]
        }]
      ]
    }
  },
}

This has treated me very well so far, but it seems incompatible with the final version of this feature? (The originally proposed version seemed much more amenable, but alas was cut.) I'm running tests via karma, so have a karma config available if that helps.

[bcoe]

@ZebraFlesh do you use nyc for running reports? if so, you can use babel-plugin-istanbul + nyc to report on untested files, like so:

nyc --all your-test-runner.

[bcoe]

@marco-ramirez have any advice for @ZebraFlesh?

[ZebraFlesh]

@bcoe Nope, I'm using karma.

[bcoe]

@ZebraFlesh I think you'll need to do something similar to nyc:

https://github.com/istanbuljs/nyc/blob/master/index.js#L235

or, @marco-ramirez's gulp plugin:

https://github.com/marco-ramirez/babel-plugin-istanbul-include-untested-example/blob/e2a8fabe697f8e1c5b7042eff43d598633ee1a95/task/test.js#L34

In which you walk all files, and collect the coverage report -- if you find a good approach for Karma, I'd love to add it to the istanbul website.

@marco-ramirez, likewise, it would be good to add a tutorial with your gulp example to the website.

[paul-sachs]

I am also having a similar issue with this project here:

https://github.com/psachs21/web-boilerplate

I run the coverage but it only reports on coverage of the tested files, not the untested files. I am using ava with nyc.

Any ideas?

[marco-ramirez]

@ZebraFlesh have you tried the karma-coverage plugin? It uses istanbul and seems to have an option to include all source files in the coverage reports.

[marco-ramirez]

@psachs21, @bcoe described how to use nyc to generate reports for all source files.

https://github.com/istanbuljs/babel-plugin-istanbul/issues/105#issuecomment-301279352

[ZebraFlesh]

I had not seen that, thank you! I'm a bit over whelmed at the moment, so will try it in a day or two.

[paul-sachs]

@marco-ramirez my example repo does everything @bcoe mentioned, it still does not work. It is using istanbul in it's babelrc file. I call nyc --all ava as suggested. And untested files are still not shown in the coverage report.

[ZebraFlesh]

So I gave that includeAllSources options on the karma-coverage plugin a try, but didn't have any luck. It seems that option has been present in the plugin for a while, but has been unreliable. So I'm back to square one on how to pass an onCover function to karma.

[ZebraFlesh]

I think I've figured out how to cover all files from a karma perspective, but I'm not using the onCover function and in general it feels like I'm using a really big hammer instead of more graceful tools. But it works. The general approach is to have the test runner load all applicable files (for however you define "applicable files") which in turn causes webpack to transpile them via babel and then instrument them via istanbul.

Am I missing something? Is onCover just not applicable in a karma environment? Or am I doing something wrong?

Previously, my karma index.js looked like this:

// Concatenates all tests (*.spec.js) so they share common code via Webpack
const testsContext = require.context('.', true, /\.spec\.js$/);
testsContext.keys().forEach(testsContext);

For simple cases, I changed it to the following:

// Concatenates all tests (*.spec.js) so they share common code via Webpack
const testsContext = require.context('.', true, /\.spec\.js$/);
testsContext.keys().forEach(testsContext);

// Require all source files for code coverage purposes.
const componentsContext = require.context('../src/', true, /\.js$/);
componentsContext.keys().forEach(componentsContext);

For more complex cases, it's possible to filter the webpack dynamic context:

// Concatenates all tests (*.spec.js) so they share common code via Webpack
const testsContext = require.context('.', true, /\.spec\.js$/);
testsContext.keys().forEach(testsContext);

// Require all source files for code coverage purposes.
const componentsContext = require.context('../src/', true, /\.js$/);
componentsContext.keys().filter(key => {
  // ...custom filtering logic here...
}).forEach(componentsContext);

[bcoe]

@ZebraFlesh any follow up work you can think of from this issue? it sounds like you did eventually get karma instrumenting all files?

Could we add documentation on the istanbul website? could updates be made to karma or Istanbul to make things easier?

[ZebraFlesh]

Correct, I did eventually get karma to instrument all of the files through istanbul. Propagating the "simple case" documentation I outlined above would probably be helpful. I'm not sure if there's anything beyond that to be done for either karma or istanbul -- there's no simple solution to the problem, so you can't just solve it with a flag.

Expounding a bit: when you run your tests, they're going to inherently pull in just their dependencies. (In this context, I'm using "dependency" to describe an ES6 module import tree.) For example, I have A.spec.js which imports A.js; A.js imports B.js and C.js from my source tree plus Z.js from node_modules. D.js from my source tree is not part of the import tree, so istanbul would never have the opportunity to cover it. Using the method outlined above, I can add D.js to karma context so now istanbul will see it. Using standard istanbul configurations, I can exclude node_modules so Z.js would correctly be excluded from coverage reports. But what if there's an E.js that I don't want to or can't pull in? (In my case, E.js is legacy code written in a different style and I'm not sure it would even successfully run under karma.) That's where that "complex case" I outlined comes in and you have to perform custom filtering logic.

Ultimately software projects are messy, but karma has the necessary hooks to deal with it and make things work with istanbul. It was just a bit of work to put the pieces together.

[kopach]

For those, who use Karma + Istanbul – take a look on this plugin https://github.com/kopach/karma-sabarivka-reporter. Configuration is simple like that:

reporters: [
  // ...
  'sabarivka'
  // ...
],
coverageReporter: {
    include: 'src/**/!(*.spec|*.module|environment*).ts',
    exclude: 'src/main.ts',
    // ...
},

It works, as already mentioned above – by scanning your project files and adding them to coverage report.