Open jhiester opened 3 years ago
Example setup for code coverage with Create-React-App, in cypress/plugins/index.js:
// Component / storybook testing
import injectDevServer from '@cypress/react/plugins/react-scripts';
import findReactScriptsWebpackConfig from '@cypress/react/plugins/react-scripts/findReactScriptsWebpackConfig';
import { startDevServer } from '@cypress/webpack-dev-server';
// Code coverage
import codeCoverageTask from '@cypress/code-coverage/task';
// This function is called when a project is opened or re-opened (e.g. due to
// the project's config changing)
/**
* @type {Cypress.PluginConfig}
*/
// eslint-disable-next-line no-unused-vars
module.exports = (on, config) => {
// `on` is used to hook into various events Cypress emits
// `config` is the resolved Cypress config
// Tell Cypress to report on code coverage
codeCoverageTask(on, config);
// Unit/Component testing web dev server setup for instrumenting code coverage
// For integration testing, this is done in CRA start command
if (config.testingType === 'component') {
injectDevServer(on, config);
const webpackConfig = findReactScriptsWebpackConfig(config, {
webpackConfigPath: 'react-scripts/config/webpack.config',
});
const rules = webpackConfig.module.rules.find((rule) => !!rule.oneOf).oneOf;
const babelRule = rules.find((rule) => /babel-loader/.test(rule.loader));
babelRule.options.plugins.push(require.resolve('babel-plugin-istanbul'));
on('dev-server:start', (options) => {
return startDevServer({ options, webpackConfig });
});
}
return config;
};
Hello @jhiester
I believe there could be two subjects to cover here:
On that front there is good news: If you run the e2e job and the component job successively, their test coverage going in the same folder, istanbul is smart and will merge the two reports automagically.
If this does not answer your question, feel free to respond below. If it does, you can close this issue.
Thank you in advance.
Hey @yann-combarnous, I have similar conf and it doesn't collect coverage for CT, for E2E it does it but after running cypress run-ct
it doesn't generate anything.
Hello @Dorious
What @yann-combarnous is doing above should be extracting the coverage in Component testing.
But this line
injectDevServer(on, config);
Will do behind the scenes the same as these lines.
on('dev-server:start', (options) => {
return startDevServer({ options, webpackConfig });
});
They will be redundant.
The first one will inject the server without setting up any coverage. The second one will inject and have coverage.
We probably should check that in the processing and send a warning.
Did you see that?
@elevatebart I don't have injectDevServer because I don't use react-scripts. My cypress plugins/index.ts
/// <reference types="cypress" />
import browserify from '@cypress/browserify-preprocessor';
import task from '@cypress/code-coverage/task';
import { initPlugin } from 'cypress-plugin-snapshots/plugin';
import { startDevServer } from '@cypress/webpack-dev-server';
import webpackConfig from '../../webpack.cypress';
/**
* @type {Cypress.PluginConfig}
*/
const configurePlugins: Cypress.PluginConfig = (on, config) => {
task(on, config);
initPlugin(on, config);
on(
'file:preprocessor',
browserify({
typescript: require.resolve('typescript'),
})
);
if (config.testingType === 'component') {
// console.log(webpackConfig.module.rules[4].use[0].options);
on('dev-server:start', (options) => startDevServer({ options, webpackConfig }));
}
return config;
};
export default configurePlugins;
The console log is there to check is istanbul
there and it is.
Also I see that the coverage is there each test:
But it doesn't push it into
coverage
dir. In integration tests (cypress run
) it works.
@elevatebart ok I figured out what was wrong in my case, in support/
I have separate component.ts
and we need to put import '@cypress/code-coverage'
:) now coverage is collected properly. @jhiester maybe that the trick for you too.
Lovely !!!
Code coverage implementation might be a good "recipe" to add to our docs.
To get coverage, you need:
plugins/index.js
, instrumentation of your code (using babel or another istanbul plugin) in the config you pass to the dev server (that is always the hard part).plugins/index.js
implement the code coverage task with the proper task.
const installCoverageTask = require("@cypress/code-coverage/task");
module.exports = (on, config) => {
installCoverageTask(on, config)
}
support/index.js
import the code-coverage implementation to run this task when needed.
import '@cypress/code-coverage/support'
@elevatebart I have followed all the steps mentioned in the repo. But the instrumentation is not happening at all in my case. we use the same react scripts.
@vrknetha What is your stack? Webpack? vite? vue? react?
@vrknetha just re-read your answer... Sorry about that, you already said you are indeed using react-scripts.
Can you share your setup?
A repository would be even better, but I understand not everyone can take this time.
Same thing here, CT code coverage is not working. CRA + Typescript. What finally made it work I added
{
test: /\.(tsx|ts)?$/,
include: [SRC_PATH],
use: [{
loader: 'istanbul-instrumenter-loader',
options: { esModules: true }
}]
}
just before my typescript loader, in webpack config for
on('dev-server:start', options => startDevServer({
options,
webpackConfig
}));
This is what I have:
const webpackConfig = {
module: {
rules: [{
test: /\.(tsx|ts)?$/,
include: [SRC_PATH],
use: [{
loader: 'istanbul-instrumenter-loader',
options: { esModules: true }
}]
}, {
test: /\.(tsx|ts)?$/,
include: [SRC_PATH, CYPRESS_PATH, CYPRESS_NODE_PATH],
use: [{
loader: 'awesome-typescript-loader',
options: {
configFileName: CONFIG_PATH
}
}]
}]
},
resolve: {
extensions: ['.tsx', '.ts', '.js', '.json']
}
};
on('dev-server:start', options => startDevServer({
options,
webpackConfig
}));
And it works for me like this... Hope it helps someone.
@yeyalab that really helps. You rock. @cowboy can you check and add this to the examples when you get a chance?
Yeah, adding code coverage is absolutely on my list!
Thanks @elevatebart - this repo helped me a bunch: https://github.com/elevatebart/cy-ct-cra
I'm not currently using integration testing, and was struggling to find resources that demo'd code coverage for a CRA app using only component testing.
In case anyone runs into the same issue, the saving grace for me was in /cypress/plugins/index.js
, importing "@cypress/instrument-cra"
. All other resources I'd found up until now said to add -r @cypress/instrument-cra
to the yarn start
script. This looks like it's for folks using integration testing though.
My final /cypress/plugins/index.js
:
/// <reference types="cypress" />
require("@cypress/instrument-cra");
const injectCraDevServer = require("@cypress/react/plugins/react-scripts");
const installCoverageTask = require("@cypress/code-coverage/task");
/**
* @type {Cypress.PluginConfig}
*/
module.exports = (on, config) => {
// `on` is used to hook into various events Cypress emits
// `config` is the resolved Cypress config
installCoverageTask(on, config);
if (config.testingType === "component") {
injectCraDevServer(on, config);
}
return config;
};
My package.json
:
"devDependencies": {
+ "@cypress/code-coverage": "^3.9.10",
+ "@cypress/instrument-cra": "^1.4.0",
+ "@cypress/react": "^5.9.4",
+ "@cypress/webpack-dev-server": "^1.4.0",
+ "@testing-library/cypress": "^8.0.0",
+ "cypress": "^8.3.0",
+ "html-webpack-plugin": "4",
One thing I have not figured out is that I want to have nyc report on all files, which is why I used in package.json:
"nyc": { "all": true, "report-dir": "coverage", "reporter": [ "text-summary", "json", "json-summary", "lcov" ], "include": [ "src/**/*.{js,jsx,mjs,ts,tsx}" ],
But it seems that for component testing, not all files are included in coverage percentage. Any idea how to have component testing honor the nyc setup for scope @elevatebart ?
@elevatebart you mentioned code coverage reports should be merged automagically, but I am not finding this true (or at least not in how I have it setup currently). I am not able to run both the cypress e2e runner AND the cypress component test runner at the same time. I run one after the other and whichever one run's lasts is whose coverage report is left in the /coverage
folder. Is there anything one needs to do to merge the reports?
Hello @charleshimmer,
Great question indeed.
I did not dig into it too much yet, I just went with the istambul-combine
recommendations.
https://github.com/jamestalmage/istanbul-combine/issues/2
Let me see if I can find a read documentation/tutorial using istanbul report
Update: report your coverage into separate json files then use nyc merge
or nyc report
to combine them.
I though it worked on my repo. I might have miscalculated.
Some help: https://github.com/istanbuljs/nyc#what-about-nyc-merge
@elevatebart , now trying with Vite+React to get code coverage for component testing, but getting some issues.
Using vite-plugin-istanbul 2.2.2, Vite 2.7.0-beta.10, @vitejs/plugin-react 1.1.0, and Cypress 8.7.0.
vite.config.js:
plugins: [
react(),
istanbul({
include: ['src/**/*.{js,jsx,mjs,ts,tsx}'],
exclude: [
'coverage',
'node_modules',
'src/**/*.{types,stories,spec,faker,test}.{js,jsx,mjs,ts,tsx}',
],
extension: ['.js', '.ts', '.tsx'],
requireEnv: true,
})
cypress/plugins/index.js
import { startDevServer } from '@cypress/vite-dev-server';
import codeCoverageTask from '@cypress/code-coverage/task';
module.exports = (on, config) => {
// `on` is used to hook into various events Cypress emits
// `config` is the resolved Cypress config
// Tell Cypress to report on code coverage
codeCoverageTask(on, config);
if (config.testingType === 'component') {
on('dev-server:start', (options) => {
return startDevServer({
options,
viteConfig: {
configFile: path.resolve(__dirname, '..', '..', 'vite.config.js'),
},
});
});
}
When running tests, tests do not run in UI. Error in Cypress UI is:
The following error originated from your test code, not from Cypress.
> Unexpected identifier
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.
at Object.runScripts (http://localhost:3000/__cypress/runner/cypress_runner.js:194369:63)
at $Cypress.onSpecWindow (http://localhost:3000/__cypress/runner/cypress_runner.js:183210:76)
at <unknown> (http://localhost:3000/__cypress/src/@fs//Users/xyz/Code/cbre-360-portal/node_modules/@cypress/vite-dev-server/client/initCypressTests.js:22:18)
In addition, Vite is watching coverage folder, getting the following line updated for all files there:
11:18:14 [vite] page reload coverage/lcov-report/src/constants/icons.ts.html
When I just changed vite.config.js and disabled Istanbul plugin, component test just run fine.
Any idea on how to get this to work with Vite + React?
In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!
https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22
What is your folder structure? We saw the same behavior and it almost always had to do with how we're including/excluding files.
In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!
https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22
Very helpful, thank you! Found my issue: one "process.env.NODE_ENV" left in my code. Interestingly, it only crashed with Vite code coverage plugin and Cypress. Vite build and dev server ran fine on their own.
@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.
function cov_26q8djqde0() {
...
var coverageData = {
inputSourceMap: {
version: 3,
sourcesContent: ["export const reproduction = (item: any) => {\n if ("development" !== \"production\") {\n console.log(\"TEST\");\n }\n\n return \"\";\n};\n"],
},
};
}
I also see https://github.com/iFaxity/vite-plugin-istanbul/issues/8 has been raised.
@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.
function cov_26q8djqde0() { ... var coverageData = { inputSourceMap: { version: 3, sourcesContent: ["export const reproduction = (item: any) => {\n if ("development" !== \"production\") {\n console.log(\"TEST\");\n }\n\n return \"\";\n};\n"], }, }; }
I also see iFaxity/vite-plugin-istanbul#8 has been raised.
Replace process.env.NODE_ENV in your code by import.meta.env.MODE. See https://vitejs.dev/guide/env-and-mode.html#env-variables .
Thanks for the quick reply. Sadly I don't think that will work for me since we're building with more than vite.
If anyone is interested, we just published oob code coverage support for Cypress + TS + Vite + Quasar, check it out: https://github.com/quasarframework/quasar-testing/releases/tag/e2e-cypress-v4.1.0
This commit can be used to replicate the setup on other frameworks/tools too, as I annotate many details in the readme
@yann-combarnous How did you end up solving this? I'm seeing the same error due to process.env.NODE_ENV being replaced in the vite-plugin-istanbul sourcesContent output.
function cov_26q8djqde0() { ... var coverageData = { inputSourceMap: { version: 3, sourcesContent: ["export const reproduction = (item: any) => {\n if ("development" !== \"production\") {\n console.log(\"TEST\");\n }\n\n return \"\";\n};\n"], }, }; }
I also see iFaxity/vite-plugin-istanbul#8 has been raised.
Just to provide an update. I finally got around to investigating this and raised a fix that was released in vite-plugin-istanbul@2.7.3
Has anyone got a working example of vue 3 + vite working with coverage?
I have installed the vite plugin with the following config options:
istanbul({
include: 'src/*',
exclude: ['node_modules'],
extension: ['.js', '.ts', '.vue'],
/**
* This allows us to omit the INSTRUMENT_BUILD env variable when running the production build via
* npm run build.
* More details below.
*/
requireEnv: false,
/**
* If forceBuildInstrument is set to true, this will add coverage instrumentation to the
* built dist files and allow the reporter to collect coverage from the (built files).
* However, when forceBuildInstrument is set to true, it will not collect coverage from
* running against the dev server: e.g. npm run dev.
*
* To allow collecting coverage from running cypress against the dev server as well as the
* preview server (built files), we use an env variable, INSTRUMENT_BUILD, to set
* forceBuildInstrument to true when running against the preview server via the
* instrument-build npm script.
*
* When you run `npm run build`, the INSTRUMENT_BUILD env variable is omitted from the npm
* script which will result in forceBuildInstrument being set to false, ensuring your
* dist/built files for production do not include coverage instrumentation code.
*/
forceBuildInstrument: Boolean(process.env.INSTRUMENT_BUILD),
})
Cypress config:
component: {
devServer: {
framework: 'vue',
bundler: 'vite',
},
screenshotOnRunFailure: false,
video: false,
},
And my package.json commands looks like:
"instrument-build": "cross-env INSTRUMENT_BUILD=true vite build",
"test:component": "npm run instrument-build && cypress run --component --reporter spec",
I'm using cypress 10.4.0 & cypress coverage 3.10.0. Any help would be greatly appreciated
I think we need a definitive guide on coverage. Some resources (other than the ones here):
Hey team! Please add your planning poker estimate with Zenhub @astone123 @marktnoonan @mike-plummer @warrensplayer @ZachJW34
In case it helps, we've been working on getting code coverage happening over here. Hopefully we'll get some of these PRs merged in soon!
https://github.com/cypress-io/cypress-component-testing-examples/pulls?q=is%3Apr+%22code+coverage%22
Those are very useful, because we need framework x bundler many recipes.
Yep, we are picking this up in our next block of work, Feb 28->Ma 14, the goal is just to document common combinations and how it works in general, so people with less common combos can configure it, too.
Yep, we are picking this up in our next block of work, Feb 28->Ma 14, the goal is just to document common combinations and how it works in general, so people with less common combos can configure it, too.
I updated the CCTDD book with the instructions for the Vite version of combined code coverage : https://app.gitbook.com/s/jK1ARkRsP5OQ6ygMk6el/ch30-appendix/combined-code-coverage#addendum-code-coverage-with-vite-instead-of-webpack .
Here's the app & working combined code cov https://github.com/muratkeremozcan/tour-of-heroes-react-vite-cypress-ts.
Hey team! Please add your planning poker estimate with Zenhub @astone123 @marktnoonan @mike-plummer @warrensplayer @jordanpowell88
Description
Not clear on how to do code coverage for component testing.
Why is this needed?
It has been really helpful to me and my organization to be able to use the code coverage reports from e2e/integration/unit tests to guide our testing and development. Having component level coverage tests would help in a similar way.
More Info
Acceptance Criteria