Document configuring code coverage for component tests (webpack, vite)

jhiester commented 3 years ago

Description

Not clear on how to do code coverage for component testing.

Why is this needed?

It has been really helpful to me and my organization to be able to use the code coverage reports from e2e/integration/unit tests to guide our testing and development. Having component level coverage tests would help in a similar way.

More Info

Comment: https://github.com/cypress-io/cypress-documentation/issues/5519
Official Docs: https://docs.cypress.io/guides/tooling/code-coverage

Acceptance Criteria

Document how to set up Code Coverage for Webpack and Vite with simple example repositories
See this issue for lots of info - ideally we should document doing code coverage w/ Webpack and Vite workflows, and provide 2 example repos.
If we can add something to docs, that might be good - we've got some info on E2E, but not much for CT (although process is mostly the same): https://docs.cypress.io/guides/tooling/code-coverage

yann-combarnous commented 3 years ago

Example setup for code coverage with Create-React-App, in cypress/plugins/index.js:

// Component / storybook testing
import injectDevServer from '@cypress/react/plugins/react-scripts';
import findReactScriptsWebpackConfig from '@cypress/react/plugins/react-scripts/findReactScriptsWebpackConfig';
import { startDevServer } from '@cypress/webpack-dev-server';
// Code coverage
import codeCoverageTask from '@cypress/code-coverage/task';

// This function is called when a project is opened or re-opened (e.g. due to
// the project's config changing)

/**
 * @type {Cypress.PluginConfig}
 */
// eslint-disable-next-line no-unused-vars
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  // Tell Cypress to report on code coverage
  codeCoverageTask(on, config);

  // Unit/Component testing web dev server setup for instrumenting code coverage
  // For integration testing, this is done in CRA start command
  if (config.testingType === 'component') {
    injectDevServer(on, config);
    const webpackConfig = findReactScriptsWebpackConfig(config, {
      webpackConfigPath: 'react-scripts/config/webpack.config',
    });
    const rules = webpackConfig.module.rules.find((rule) => !!rule.oneOf).oneOf;
    const babelRule = rules.find((rule) => /babel-loader/.test(rule.loader));
    babelRule.options.plugins.push(require.resolve('babel-plugin-istanbul'));

    on('dev-server:start', (options) => {
      return startDevServer({ options, webpackConfig });
    });
  }
  return config;
};

elevatebart commented 3 years ago

Hello @jhiester

I believe there could be two subjects to cover here:

How to get component testing code coverage

If you are using create react app see in this repo https://github.com/elevatebart/cy-ct-cra
If you are using Vue see this example https://github.com/cypress-io/cypress/tree/npm/vue/v2/npm/vue/examples/code-coverage

How to merge it with e2e testing coverage

On that front there is good news: If you run the e2e job and the component job successively, their test coverage going in the same folder, istanbul is smart and will merge the two reports automagically.

If this does not answer your question, feel free to respond below. If it does, you can close this issue.

Thank you in advance.

Dorious commented 3 years ago

Hey @yann-combarnous, I have similar conf and it doesn't collect coverage for CT, for E2E it does it but after running cypress run-ct it doesn't generate anything.

elevatebart commented 3 years ago

Hello @Dorious

What @yann-combarnous is doing above should be extracting the coverage in Component testing.

But this line

injectDevServer(on, config);

Will do behind the scenes the same as these lines.

on('dev-server:start', (options) => {
  return startDevServer({ options, webpackConfig });
});

They will be redundant.

The first one will inject the server without setting up any coverage. The second one will inject and have coverage.

We probably should check that in the processing and send a warning.

Did you see that?

Dorious commented 3 years ago

@elevatebart I don't have injectDevServer because I don't use react-scripts. My cypress plugins/index.ts

/// <reference types="cypress" />

import browserify from '@cypress/browserify-preprocessor';
import task from '@cypress/code-coverage/task';
import { initPlugin } from 'cypress-plugin-snapshots/plugin';
import { startDevServer } from '@cypress/webpack-dev-server';

import webpackConfig from '../../webpack.cypress';

/**
 * @type {Cypress.PluginConfig}
 */
const configurePlugins: Cypress.PluginConfig = (on, config) => {
  task(on, config);
  initPlugin(on, config);

  on(
    'file:preprocessor',
    browserify({
      typescript: require.resolve('typescript'),
    })
  );

  if (config.testingType === 'component') {
    // console.log(webpackConfig.module.rules[4].use[0].options);
    on('dev-server:start', (options) => startDevServer({ options, webpackConfig }));
  }

  return config;
};

export default configurePlugins;

The console log is there to check is istanbul there and it is.

Dorious commented 3 years ago

Also I see that the coverage is there each test: But it doesn't push it into coverage dir. In integration tests (cypress run) it works.

Dorious commented 3 years ago

@elevatebart ok I figured out what was wrong in my case, in support/ I have separate component.ts and we need to put import '@cypress/code-coverage' :) now coverage is collected properly. @jhiester maybe that the trick for you too.

elevatebart commented 3 years ago

Lovely !!!

Code coverage implementation might be a good "recipe" to add to our docs.

To get coverage, you need:

In plugins/index.js, instrumentation of your code (using babel or another istanbul plugin) in the config you pass to the dev server (that is always the hard part).

In plugins/index.js implement the code coverage task with the proper task.

const installCoverageTask = require("@cypress/code-coverage/task");
module.exports = (on, config) => {
installCoverageTask(on, config)
}

In support/index.js import the code-coverage implementation to run this task when needed.
```
import '@cypress/code-coverage/support'
```

vrknetha commented 3 years ago

@elevatebart I have followed all the steps mentioned in the repo. But the instrumentation is not happening at all in my case. we use the same react scripts.

elevatebart commented 3 years ago

@vrknetha What is your stack? Webpack? vite? vue? react?

elevatebart commented 3 years ago

@vrknetha just re-read your answer... Sorry about that, you already said you are indeed using react-scripts.

Can you share your setup?

package.json
cypress.json
cypress/plugins & cypress/support

A repository would be even better, but I understand not everyone can take this time.

jovancacvetkovic commented 3 years ago

Same thing here, CT code coverage is not working. CRA + Typescript. What finally made it work I added

{
        test: /\.(tsx|ts)?$/,
        include: [SRC_PATH],
        use: [{
          loader: 'istanbul-instrumenter-loader',
          options: { esModules: true }
        }]
      }

just before my typescript loader, in webpack config for

on('dev-server:start', options => startDevServer({
    options,
    webpackConfig
  }));

This is what I have:

const webpackConfig = {
    module: {
      rules: [{
        test: /\.(tsx|ts)?$/,
        include: [SRC_PATH],
        use: [{
          loader: 'istanbul-instrumenter-loader',
          options: { esModules: true }
        }]
      }, {
        test: /\.(tsx|ts)?$/,
        include: [SRC_PATH, CYPRESS_PATH, CYPRESS_NODE_PATH],
        use: [{
          loader: 'awesome-typescript-loader',
          options: {
            configFileName: CONFIG_PATH
          }
        }]
      }]
    },
    resolve: {
      extensions: ['.tsx', '.ts', '.js', '.json']
    }
  };

  on('dev-server:start', options => startDevServer({
    options,
    webpackConfig
  }));

And it works for me like this... Hope it helps someone.

elevatebart commented 3 years ago

@yeyalab that really helps. You rock. @cowboy can you check and add this to the examples when you get a chance?

cowboy commented 3 years ago

Yeah, adding code coverage is absolutely on my list!

chasinhues commented 3 years ago

Thanks @elevatebart - this repo helped me a bunch: https://github.com/elevatebart/cy-ct-cra

I'm not currently using integration testing, and was struggling to find resources that demo'd code coverage for a CRA app using only component testing.

In case anyone runs into the same issue, the saving grace for me was in /cypress/plugins/index.js, importing "@cypress/instrument-cra". All other resources I'd found up until now said to add -r @cypress/instrument-cra to the yarn start script. This looks like it's for folks using integration testing though.

My final /cypress/plugins/index.js:

/// <reference types="cypress" />

require("@cypress/instrument-cra");
const injectCraDevServer = require("@cypress/react/plugins/react-scripts");
const installCoverageTask = require("@cypress/code-coverage/task");

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
    // `on` is used to hook into various events Cypress emits
    // `config` is the resolved Cypress config

    installCoverageTask(on, config);

    if (config.testingType === "component") {
        injectCraDevServer(on, config);
    }

    return config;
};

My package.json:

     "devDependencies": {
+        "@cypress/code-coverage": "^3.9.10",
+        "@cypress/instrument-cra": "^1.4.0",
+        "@cypress/react": "^5.9.4",
+        "@cypress/webpack-dev-server": "^1.4.0",
+        "@testing-library/cypress": "^8.0.0",
+        "cypress": "^8.3.0",
+        "html-webpack-plugin": "4",

yann-combarnous commented 3 years ago

One thing I have not figured out is that I want to have nyc report on all files, which is why I used in package.json: "nyc": { "all": true, "report-dir": "coverage", "reporter": [ "text-summary", "json", "json-summary", "lcov" ], "include": [ "src/**/*.{js,jsx,mjs,ts,tsx}" ],

But it seems that for component testing, not all files are included in coverage percentage. Any idea how to have component testing honor the nyc setup for scope @elevatebart ?

charleshimmer commented 2 years ago

@elevatebart you mentioned code coverage reports should be merged automagically, but I am not finding this true (or at least not in how I have it setup currently). I am not able to run both the cypress e2e runner AND the cypress component test runner at the same time. I run one after the other and whichever one run's lasts is whose coverage report is left in the /coverage folder. Is there anything one needs to do to merge the reports?

elevatebart commented 2 years ago

Hello @charleshimmer,

Great question indeed. I did not dig into it too much yet, I just went with the istambul-combine recommendations.

https://github.com/jamestalmage/istanbul-combine/issues/2

Let me see if I can find a read documentation/tutorial using istanbul report

elevatebart commented 2 years ago

Update: report your coverage into separate json files then use nyc merge or nyc report to combine them. I though it worked on my repo. I might have miscalculated.

Some help: https://github.com/istanbuljs/nyc#what-about-nyc-merge

yann-combarnous commented 2 years ago

@elevatebart , now trying with Vite+React to get code coverage for component testing, but getting some issues.

Using vite-plugin-istanbul 2.2.2, Vite 2.7.0-beta.10, @vitejs/plugin-react 1.1.0, and Cypress 8.7.0.

vite.config.js:

    plugins: [
      react(),
      istanbul({
        include: ['src/**/*.{js,jsx,mjs,ts,tsx}'],
        exclude: [
          'coverage',
          'node_modules',
          'src/**/*.{types,stories,spec,faker,test}.{js,jsx,mjs,ts,tsx}',
        ],
        extension: ['.js', '.ts', '.tsx'],
        requireEnv: true,
     })

cypress/plugins/index.js

import { startDevServer } from '@cypress/vite-dev-server';
import codeCoverageTask from '@cypress/code-coverage/task';

module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  // Tell Cypress to report on code coverage
  codeCoverageTask(on, config);

  if (config.testingType === 'component') {
    on('dev-server:start', (options) => {
      return startDevServer({
        options,
        viteConfig: {
          configFile: path.resolve(__dirname, '..', '..', 'vite.config.js'),
        },
      });
    });
  }

When running tests, tests do not run in UI. Error in Cypress UI is:

The following error originated from your test code, not from Cypress.
  > Unexpected identifier
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.

at Object.runScripts (http://localhost:3000/__cypress/runner/cypress_runner.js:194369:63)
at $Cypress.onSpecWindow (http://localhost:3000/__cypress/runner/cypress_runner.js:183210:76)
at <unknown> (http://localhost:3000/__cypress/src/@fs//Users/xyz/Code/cbre-360-portal/node_modules/@cypress/vite-dev-server/client/initCypressTests.js:22:18)

In addition, Vite is watching coverage folder, getting the following line updated for all files there:

11:18:14 [vite] page reload coverage/lcov-report/src/constants/icons.ts.html

When I just changed vite.config.js and disabled Istanbul plugin, component test just run fine.

Any idea on how to get this to work with Vite + React?

cowboy commented 2 years ago

In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!

https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22

samtsai commented 2 years ago

What is your folder structure? We saw the same behavior and it almost always had to do with how we're including/excluding files.

yann-combarnous commented 2 years ago

In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!

https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22

Very helpful, thank you! Found my issue: one "process.env.NODE_ENV" left in my code. Interestingly, it only crashed with Vite code coverage plugin and Cypress. Vite build and dev server ran fine on their own.

joshwooding commented 2 years ago

@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.

function cov_26q8djqde0() {
    ...
    var coverageData = {
        inputSourceMap: {
            version: 3,
            sourcesContent: ["export const reproduction = (item: any) => {\n  if ("development" !== \"production\") {\n    console.log(\"TEST\");\n  }\n\n  return \"\";\n};\n"],
        },
    };

}

I also see https://github.com/iFaxity/vite-plugin-istanbul/issues/8 has been raised.

yann-combarnous commented 2 years ago

@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.
function cov_26q8djqde0() {
    ...
    var coverageData = {
        inputSourceMap: {
            version: 3,
            sourcesContent: ["export const reproduction = (item: any) => {\n  if ("development" !== \"production\") {\n    console.log(\"TEST\");\n  }\n\n  return \"\";\n};\n"],
        },
    };

}
I also see iFaxity/vite-plugin-istanbul#8 has been raised.

Replace process.env.NODE_ENV in your code by import.meta.env.MODE. See https://vitejs.dev/guide/env-and-mode.html#env-variables .

joshwooding commented 2 years ago

Thanks for the quick reply. Sadly I don't think that will work for me since we're building with more than vite.

IlCallo commented 2 years ago

If anyone is interested, we just published oob code coverage support for Cypress + TS + Vite + Quasar, check it out: https://github.com/quasarframework/quasar-testing/releases/tag/e2e-cypress-v4.1.0

This commit can be used to replicate the setup on other frameworks/tools too, as I annotate many details in the readme

joshwooding commented 2 years ago

@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.
function cov_26q8djqde0() {
    ...
    var coverageData = {
        inputSourceMap: {
            version: 3,
            sourcesContent: ["export const reproduction = (item: any) => {\n  if ("development" !== \"production\") {\n    console.log(\"TEST\");\n  }\n\n  return \"\";\n};\n"],
        },
    };

}
I also see iFaxity/vite-plugin-istanbul#8 has been raised.

Just to provide an update. I finally got around to investigating this and raised a fix that was released in vite-plugin-istanbul@2.7.3

Ancient-Dragon commented 2 years ago

Has anyone got a working example of vue 3 + vite working with coverage?

I have installed the vite plugin with the following config options:

istanbul({
        include: 'src/*',
        exclude: ['node_modules'],
        extension: ['.js', '.ts', '.vue'],
        /**
         * This allows us to omit the INSTRUMENT_BUILD env variable when running the production build via
         * npm run build.
         * More details below.
         */
        requireEnv: false,
        /**
         * If forceBuildInstrument is set to true, this will add coverage instrumentation to the
         * built dist files and allow the reporter to collect coverage from the (built files).
         * However, when forceBuildInstrument is set to true, it will not collect coverage from
         * running against the dev server: e.g. npm run dev.
         *
         * To allow collecting coverage from running cypress against the dev server as well as the
         * preview server (built files), we use an env variable, INSTRUMENT_BUILD, to set
         * forceBuildInstrument to true when running against the preview server via the
         * instrument-build npm script.
         *
         * When you run `npm run build`, the INSTRUMENT_BUILD env variable is omitted from the npm
         * script which will result in forceBuildInstrument being set to false, ensuring your
         * dist/built files for production do not include coverage instrumentation code.
         */
        forceBuildInstrument: Boolean(process.env.INSTRUMENT_BUILD),
      })

Cypress config:

  component: {
    devServer: {
      framework: 'vue',
      bundler: 'vite',
    },
    screenshotOnRunFailure: false,
    video: false,
  },

And my package.json commands looks like:

    "instrument-build": "cross-env INSTRUMENT_BUILD=true vite build",
    "test:component": "npm run instrument-build && cypress run --component --reporter spec",

I'm using cypress 10.4.0 & cypress coverage 3.10.0. Any help would be greatly appreciated

lmiller1990 commented 1 year ago

I think we need a definitive guide on coverage. Some resources (other than the ones here):

CRA + webpack: https://muratkerem.gitbook.io/cctdd/ch30-appendix/combined-code-coverage
Vite (some info floating around here) https://github.com/iFaxity/vite-plugin-istanbul/issues/22

lmiller1990 commented 1 year ago

Hey team! Please add your planning poker estimate with Zenhub @astone123 @marktnoonan @mike-plummer @warrensplayer @ZachJW34

muratkeremozcan commented 1 year ago

In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!

https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22

Those are very useful, because we need framework x bundler many recipes.

lmiller1990 commented 1 year ago

Yep, we are picking this up in our next block of work, Feb 28->Ma 14, the goal is just to document common combinations and how it works in general, so people with less common combos can configure it, too.

muratkeremozcan commented 1 year ago

Yep, we are picking this up in our next block of work, Feb 28->Ma 14, the goal is just to document common combinations and how it works in general, so people with less common combos can configure it, too.

I updated the CCTDD book with the instructions for the Vite version of combined code coverage : https://app.gitbook.com/s/jK1ARkRsP5OQ6ygMk6el/ch30-appendix/combined-code-coverage#addendum-code-coverage-with-vite-instead-of-webpack .

Here's the app & working combined code cov https://github.com/muratkeremozcan/tour-of-heroes-react-vite-cypress-ts.

lmiller1990 commented 1 year ago

Hey team! Please add your planning poker estimate with Zenhub @astone123 @marktnoonan @mike-plummer @warrensplayer @jordanpowell88

lmiller1990 commented 1 year ago

Relevent https://github.com/cypress-io/cypress-component-testing-apps/issues/16

cypress-io / cypress-documentation