Closed penx closed 4 years ago
I'm thinking this could be an issue with @cypress-io/instrument-cra
It looks like instrument-cra pushes the babel plugin istanbul plugin in to the webpack build, so perhaps Istanbul is only aware of files that are being built by webpack. In the example above, I have files in my project that aren't included in the webpack build, but I would want these to be included in the code coverage report (it is useful to utilise code coverage reports from integration tests to identify redundant files that are candidates for deletion).
Hi, I looked at the project
src/utilities
- I don't think NYC itself does anything differently. If after Cypress test I run--------------------------------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
--------------------------------|---------|----------|---------|---------|-------------------
All files | 84 | 71.43 | 66.67 | 100 |
src | 100 | 100 | 100 | 100 |
index.js | 100 | 100 | 100 | 100 |
src/application | 100 | 100 | 100 | 100 |
App.js | 100 | 100 | 100 | 100 |
src/design-system/GlobalStyles | 100 | 50 | 100 | 100 |
index.js | 100 | 50 | 100 | 100 | 21
src/design-system/Toggle | 100 | 75 | 100 | 100 |
index.js | 100 | 75 | 100 | 100 | 12
src/utilities | 60 | 100 | 20 | 100 |
useToggle.js | 60 | 100 | 20 | 100 |
--------------------------------|---------|----------|---------|---------|-------------------
I don't see additional files in the list. Have you seen empty files before?
Can't configure cypress-specific nyc settings outside of package.json
- I have closed this in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.0I don't see additional files in the list. Have you seen empty files before?
I haven't seen Cypress report on uncovered files before, if that's what you mean? But I've only just started with this.
However, if you run yarn coverage:unit
or coverage:visual-regression
on the above project you'll see that Istanbul will report on uncovered files:
--------------------------------|----------|----------|----------|----------|-------------------|
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s |
--------------------------------|----------|----------|----------|----------|-------------------|
All files | 32.26 | 14.29 | 33.33 | 25 | |
src | 0 | 100 | 100 | 0 | |
index.js | 0 | 100 | 100 | 0 | 6 |
src/application | 0 | 100 | 0 | 0 | |
App.js | 0 | 100 | 0 | 0 | 7,8 |
src/design-system/Alert | 0 | 100 | 0 | 0 | |
index.js | 0 | 100 | 0 | 0 | 3 |
src/design-system/GlobalStyles | 0 | 0 | 0 | 0 | |
index.js | 0 | 0 | 0 | 0 |4,7,12,14,15,19,20 |
src/design-system/Message | 0 | 100 | 0 | 0 | |
index.js | 0 | 100 | 0 | 0 | 3 |
src/design-system/Toggle | 0 | 0 | 0 | 0 | |
index.js | 0 | 0 | 0 | 0 | 4,5,6,9,10 |
src/utilities | 83.33 | 100 | 83.33 | 85.71 | |
addOne.js | 0 | 100 | 0 | 0 | 1 |
useToggle.js | 100 | 100 | 100 | 100 | |
--------------------------------|----------|----------|----------|----------|-------------------|
Yeah, I think you have identified the problem - the babel-plugin-istanbul instrumet-CRA is inserting is only instrumenting the files the webpack gives to it. Maybe https://github.com/cypress-io/instrument-cra/pull/75 will change its behavior, let me see if I can upgrade that module.
PS: love your project, we definitely need to add it as an example to this repo's README
PS: love your project, we definitely need to add it as an example to this repo's README
thanks! I'm working on a blog post to accompany it, but don't want to publish until I can get this thing working. I can send you a draft if you're interested
I found this which is loosely related https://github.com/istanbuljs/babel-plugin-istanbul/issues/105
Perhaps in task.js:resetCoverage() you can add something that adds all files to the coverage report as soon as it is initialised (with zero coverage) independently of create-react-app/webpack?
@penx can you try v3.5.0 - it should insert dummy placeholders for all files from package.json nyc
list (if you use all:true
option) into .nyc_output/out.json
file before generating a report. It would look something like this:
Great, that works! The uncovered files are white and don't have stats against them like number of branches and functions. One side effect of this is the lack of tests doesn't affect the overall code coverage metric. This isn't too important for me right now but perhaps something you may want to look at in the future? I could add a new feature request ticket if you like? Thanks for getting this fixed so quickly.
for your known issue Can't configure cypress-specific nyc settings outside of package.json - I have closed this in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.0
I was hoping to have a cypress specific file to put in cypress specific nyc settings, as other parts of the application may also use nyc.
e.g.
"nyc": {
"report-dir": "coverage/integration",
The report dir is a cypress specific setting.
Perhaps a nyc
or code-coverage: { nyc: {
section could be added to cypress.json?
For now, I would not want to add any more specific options to Cypress, so yes, please open a new issue about empty coverage for non-included files, ok.
On a related note, I noticed when I added "reporter": ["text", "lcov"],
to nyc
in package.json
I got the following error:
CypressError: `cy.task('coverageReport')` failed with the following error:
> Error: ENOENT: no such file or directory, open '/Users/penx/Development/storybook-code-coverage/coverage/integration/coverage-final.json'
I'll take another look later or tomorrow and open a bug report or have a go at fixing it
@penx Ha, I just pushed a fix, this was just a silly bug, I forgot existsSync(...)
here
const jsonReportFilename = join(folder, 'coverage-final.json')
if (!existsSync) {
debug('Did not find final coverage file %s', jsonReportFilename)
return
}
I've got all: true
in nyc
config. I get the dummy entries in coverage report.
I've used this approach here https://github.com/toaditoad/code-coverage-mwe with react to instrument untested/ uncovered files. A downside of this approach is that it adds to the completion time of test runs due to webpack additionalEntries
.
related issues https://github.com/istanbuljs/nyc/issues/1415 https://github.com/cypress-io/cypress/issues/17230
I've a monorepo setup with cypress installed in each of the sub-package (yarn workspaces). For the sub-packages with webpack, additionalEntries
param in webpack config solves this issue. But I'm stuck with a sub-package that doesn't have webpack, as I don't have additionalEntries
at my disposal. Is there a way to achieve this with babel setup?
Follow up to #194
Logs and screenshots
Versions
What is this plugin's version? 3.4.1
What is Cypress version? 4.4.0
What is your operating system? Mac OS X 10.15.4
What is the shell? zsh
What is the Node version? 12.14.1
What is the NPM version? 6.13.4
How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.
When running tests, if you open the web application in regular browser, and open DevTools, do you see
window.__coverage__
object? Can you paste a screenshot?Yes
.nyc_output
folder? Is there.nyc_output/out.json
file. Is it empty? Can you paste at least part of it so we can see the keys and file paths?Yes
package.json
(nyc
object) or in other NYC config filesYes, as per 'describe the bug' below
No
Describe the bug If I have the following in my package.json:
I would expect to see files that have 0% coverage in the report, but they don't appear.
Link to the repo
https://github.com/penx/storybook-code-coverage
^ at present, you will need to update this repo to 3.4.1 and remove
"report-dir": "coverage/integration",
from package.json (as this setting seems to be bugged in 3.4.1)