Test coverage in v2 addons

[SergeAstapov]

Are there any recommendations how to setup test coverage in v2 addons?

With v1 addons or apps, we could follow instructions outlined in https://github.com/ember-cli-code-coverage/ember-cli-code-coverage#installation

With v2 addons, above is not expected to work because:

1. Addon is now built separately from test-app
2. Installing ember-cli-code-coverage in the addon won't make any effect as there are no tests to run and then report/collect coverage
3. Installing ember-cli-code-coverage in the test-app won't make any effect as only app cove will get instrumented and no effect for the addon. Additionally, test-app files will be considered for coverage.

I assume, we need to instrument the addon at build time with https://www.npmjs.com/package/babel-plugin-istanbul and then we need tell ember-cli-code-coverage to report/collect coverage only for the addon code.

@runspired mentioned on Discord some of his team members did such setup. @runspired do you mind to give some pointers on this or maybe just delegate to someone to share the setup?

[NullVoxPopuli]

the approach that I've seen used is bad, in that we have a one-off build, using that babel plugin you listed, instrumenting all addon code -- which gets you coverage... but at the cost of testing against something you don't use in real projects.

Example babel config:

module.exports = function (api) {
    api.cache(true);

    return {
        presets: [['@babel/preset-typescript']],
        plugins: [
            ['@babel/plugin-transform-typescript', { allowDeclareFields: true }],
            'ember-template-imports/src/babel-plugin',
            '@embroider/addon-dev/template-colocation-plugin',
            ['@babel/plugin-proposal-decorators', { legacy: true }],
            '@babel/plugin-transform-class-properties',
            '@babel/plugin-transform-private-methods',
            'ember-concurrency/lib/babel-plugin-transform-ember-concurrency-async-tasks',
            // Enabled internally if COVERAGE=true
            ...require('ember-cli-code-coverage').buildBabelPlugin(),
        ],
    };
};

We really need an entirely different approach to coverage that doesn't require mangling code, and allows us to observe converage of code without custom builds just for coverage's sake

[runspired]

Until someone does the work to fix all the small things the chrome coverage stats API does just slightly wrong (and ensures source maps are 100% accurate) I think the transform is as good as we're going to get. I agree its not ideal though.

What's not shown here is the logic to recombine the coverage into a single report, we should publicize that somewhere.

[runspired]

also note: @wagenet did a lot of the work to make the chrome API work, just not enough to get us fully over the line. Maybe he can leave a paper trail.

[SergeAstapov]

We really need an entirely different approach to coverage that doesn't require mangling code, and allows us to observe converage of code without custom builds just for coverage's sake

for that to work, I assume we would need to tweak webpack/vite to instrument addon code when it's pulled into test-app with something like https://v4.webpack.js.org/loaders/istanbul-instrumenter-loader/

Collecting/reporting still will happen byember-cli-code-coverage I assume.

[wagenet]

I have a patched version of testem that boots up Chrome with native code coverage then dumps that to a json which I can feed into Istanbul. The only thing keeping it over the line was that Chrome seems to have a bug when your JS files get over 35mb or so 😬. I hope to revisit in the next couple months.