cypress-io / code-coverage

Saves the code coverage collected during Cypress tests
MIT License
430 stars 108 forks source link

nyc config `all:true` does not work #206

Closed penx closed 4 years ago

penx commented 4 years ago

Follow up to #194

Logs and screenshots

✗ DEBUG=code-coverage yarn coverage:integration 
yarn run v1.21.1
$ rm -rf coverage/integration && mkdir -p coverage/integration && yarn cypress run
$ /Users/penx/Development/storybook-code-coverage/node_modules/.bin/cypress run
GVA encoder info: AMD performance mode : 2 
GVA encoder info: deleteSCDMetalContext : texture cache hits: 0, misses: 0

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    4.4.0                                                                              │
  │ Browser:    Electron 80 (headless)                                                             │
  │ Specs:      1 found (spec.js)                                                                  │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  spec.js                                                                         (1 of 1)

  ✓ has a visible header (381ms)
  code-coverage parsed sent coverage +0ms
  code-coverage wrote coverage file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json +2ms
  code-coverage NYC file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json has 5 key(s) +0ms
  code-coverage 1 key /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js file path /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js +0ms
  code-coverage 2 key /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js file path /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js +0ms
  code-coverage 3 key /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js file path /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js +0ms
  code-coverage in file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json all files are not found? false +0ms
  code-coverage NYC file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json has 5 key(s) +2ms
  code-coverage combined NYC options { 'report-dir': './coverage', reporter: [ 'lcov', 'clover', 'json' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: true, all: true, include: [ 'src/**/*.js' ], exclude: [ '**/*.test.js', '**/test.js', '**/*.stories.js', '**/stories.js' ] } +0ms
  code-coverage calling NYC reporter with options { 'report-dir': '/Users/penx/Development/storybook-code-coverage/coverage', reporter: [ 'lcov', 'clover', 'json' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: true, all: true, include: [ 'src/**/*.js' ], exclude: [ '**/*.test.js', '**/test.js', '**/*.stories.js', '**/stories.js' ], tempDir: '/Users/penx/Development/storybook-code-coverage/.nyc_output' } +28ms
  code-coverage current working directory is /Users/penx/Development/storybook-code-coverage +0ms
  code-coverage after reporting, returning the report folder name /Users/penx/Development/storybook-code-coverage/coverage +56ms
  code-coverage Final coverage in /Users/penx/Development/storybook-code-coverage/coverage/coverage-final.json +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/index.js statements covered 1/1 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/application/App.js statements covered 2/2 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js statements covered 7/7 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js statements covered 5/5 +0ms
  code-coverage ⚠️ /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js statements covered 6/10 +1ms

  1 passing (505ms)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     0 seconds                                                                        │
  │ Spec Ran:     spec.js                                                                          │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /Users/penx/Development/storybook-code-coverage/cypress/vid    (0 seconds)
                          eos/spec.js.mp4                                                           

====================================================================================================

  (Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  spec.js                                  497ms        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        497ms        1        1        -        -        -  

✨  Done in 8.89s.

Versions

Yes

Screenshot 2020-04-21 at 13 04 17

Yes

Screenshot 2020-04-21 at 13 05 15

Yes, as per 'describe the bug' below

No

Describe the bug If I have the following in my package.json:

  "nyc": {
    "all": true,
    "include": ["src/**/*.js"],
    "exclude": ["**/*.test.js", "**/*.stories.js"]
  }

I would expect to see files that have 0% coverage in the report, but they don't appear.

Link to the repo

https://github.com/penx/storybook-code-coverage

^ at present, you will need to update this repo to 3.4.1 and remove "report-dir": "coverage/integration", from package.json (as this setting seems to be bugged in 3.4.1)

penx commented 4 years ago

I'm thinking this could be an issue with @cypress-io/instrument-cra

https://github.com/cypress-io/instrument-cra/blob/bf977b19c77799e8b9c9f4a4ee06ae42fe7678ac/index.js#L33

It looks like instrument-cra pushes the babel plugin istanbul plugin in to the webpack build, so perhaps Istanbul is only aware of files that are being built by webpack. In the example above, I have files in my project that aren't included in the webpack build, but I would want these to be included in the code coverage report (it is useful to utilise code coverage reports from integration tests to identify redundant files that are candidates for deletion).

bahmutov commented 4 years ago

Hi, I looked at the project

--------------------------------|---------|----------|---------|---------|-------------------
File                            | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------------------------|---------|----------|---------|---------|-------------------
All files                       |      84 |    71.43 |   66.67 |     100 |                   
 src                            |     100 |      100 |     100 |     100 |                   
  index.js                      |     100 |      100 |     100 |     100 |                   
 src/application                |     100 |      100 |     100 |     100 |                   
  App.js                        |     100 |      100 |     100 |     100 |                   
 src/design-system/GlobalStyles |     100 |       50 |     100 |     100 |                   
  index.js                      |     100 |       50 |     100 |     100 | 21                
 src/design-system/Toggle       |     100 |       75 |     100 |     100 |                   
  index.js                      |     100 |       75 |     100 |     100 | 12                
 src/utilities                  |      60 |      100 |      20 |     100 |                   
  useToggle.js                  |      60 |      100 |      20 |     100 |                   
--------------------------------|---------|----------|---------|---------|-------------------

I don't see additional files in the list. Have you seen empty files before?

penx commented 4 years ago

I don't see additional files in the list. Have you seen empty files before?

I haven't seen Cypress report on uncovered files before, if that's what you mean? But I've only just started with this.

However, if you run yarn coverage:unit or coverage:visual-regression on the above project you'll see that Istanbul will report on uncovered files:

--------------------------------|----------|----------|----------|----------|-------------------|
File                            |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
--------------------------------|----------|----------|----------|----------|-------------------|
All files                       |    32.26 |    14.29 |    33.33 |       25 |                   |
 src                            |        0 |      100 |      100 |        0 |                   |
  index.js                      |        0 |      100 |      100 |        0 |                 6 |
 src/application                |        0 |      100 |        0 |        0 |                   |
  App.js                        |        0 |      100 |        0 |        0 |               7,8 |
 src/design-system/Alert        |        0 |      100 |        0 |        0 |                   |
  index.js                      |        0 |      100 |        0 |        0 |                 3 |
 src/design-system/GlobalStyles |        0 |        0 |        0 |        0 |                   |
  index.js                      |        0 |        0 |        0 |        0 |4,7,12,14,15,19,20 |
 src/design-system/Message      |        0 |      100 |        0 |        0 |                   |
  index.js                      |        0 |      100 |        0 |        0 |                 3 |
 src/design-system/Toggle       |        0 |        0 |        0 |        0 |                   |
  index.js                      |        0 |        0 |        0 |        0 |        4,5,6,9,10 |
 src/utilities                  |    83.33 |      100 |    83.33 |    85.71 |                   |
  addOne.js                     |        0 |      100 |        0 |        0 |                 1 |
  useToggle.js                  |      100 |      100 |      100 |      100 |                   |
--------------------------------|----------|----------|----------|----------|-------------------|
bahmutov commented 4 years ago

Yeah, I think you have identified the problem - the babel-plugin-istanbul instrumet-CRA is inserting is only instrumenting the files the webpack gives to it. Maybe https://github.com/cypress-io/instrument-cra/pull/75 will change its behavior, let me see if I can upgrade that module.

PS: love your project, we definitely need to add it as an example to this repo's README

penx commented 4 years ago

PS: love your project, we definitely need to add it as an example to this repo's README

thanks! I'm working on a blog post to accompany it, but don't want to publish until I can get this thing working. I can send you a draft if you're interested

penx commented 4 years ago

I found this which is loosely related https://github.com/istanbuljs/babel-plugin-istanbul/issues/105

Perhaps in task.js:resetCoverage() you can add something that adds all files to the coverage report as soon as it is initialised (with zero coverage) independently of create-react-app/webpack?

bahmutov commented 4 years ago

@penx can you try v3.5.0 - it should insert dummy placeholders for all files from package.json nyc list (if you use all:true option) into .nyc_output/out.json file before generating a report. It would look something like this:

Screen Shot 2020-04-21 at 12 51 35 PM
penx commented 4 years ago

Great, that works! The uncovered files are white and don't have stats against them like number of branches and functions. One side effect of this is the lack of tests doesn't affect the overall code coverage metric. This isn't too important for me right now but perhaps something you may want to look at in the future? I could add a new feature request ticket if you like? Thanks for getting this fixed so quickly.

penx commented 4 years ago

for your known issue Can't configure cypress-specific nyc settings outside of package.json - I have closed this in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.0

I was hoping to have a cypress specific file to put in cypress specific nyc settings, as other parts of the application may also use nyc.

e.g.

  "nyc": {
    "report-dir": "coverage/integration",

The report dir is a cypress specific setting.

Perhaps a nyc or code-coverage: { nyc: { section could be added to cypress.json?

bahmutov commented 4 years ago

For now, I would not want to add any more specific options to Cypress, so yes, please open a new issue about empty coverage for non-included files, ok.

penx commented 4 years ago

On a related note, I noticed when I added "reporter": ["text", "lcov"], to nyc in package.json I got the following error:

     CypressError: `cy.task('coverageReport')` failed with the following error:

> Error: ENOENT: no such file or directory, open '/Users/penx/Development/storybook-code-coverage/coverage/integration/coverage-final.json'

I'll take another look later or tomorrow and open a bug report or have a go at fixing it

bahmutov commented 4 years ago

@penx Ha, I just pushed a fix, this was just a silly bug, I forgot existsSync(...) here

const jsonReportFilename = join(folder, 'coverage-final.json')
if (!existsSync) {
  debug('Did not find final coverage file %s', jsonReportFilename)
  return
}
mukherje2 commented 2 years ago

I've got all: true in nyc config. I get the dummy entries in coverage report.

I've used this approach here https://github.com/toaditoad/code-coverage-mwe with react to instrument untested/ uncovered files. A downside of this approach is that it adds to the completion time of test runs due to webpack additionalEntries.

related issues https://github.com/istanbuljs/nyc/issues/1415 https://github.com/cypress-io/cypress/issues/17230

I've a monorepo setup with cypress installed in each of the sub-package (yarn workspaces). For the sub-packages with webpack, additionalEntries param in webpack config solves this issue. But I'm stuck with a sub-package that doesn't have webpack, as I don't have additionalEntries at my disposal. Is there a way to achieve this with babel setup?