Open stahloss opened 4 years ago
This is difficult for us to support as you are not using nyc to generate any coverage. From the limited information you shared I see some version conflicts:
@cypress/code-coverage@3
and nyc@15
both use the latest versions of istanbul librariesAs for your command, nyc report combined --reporter html --reporter text
is incorrect. You need nyc report -t combined --reporter html --reporter text
.
Further you have not linked a repository as required so we do not know how you are even running the tests, what kind of configurations you are using, etc.
Thanks for your response. Yes, Jest has the istanbul instrumenter built in, so I tried to align other dependencies to that, because before the downgrade I got this error: Invalid file coverage object, missing keys, found:data
. @cypress/code-coverage doesn't instrument by itself, so I can control that as opposed to Jest.
You're right the command I posted is wrong, but I used the correct one, so it's a typo.
Yes, sorry for that, I was hoping the information I supplies would be enough, because my production setup if fairly complex with a mix of nx with angular, cypress, jest and nyc. It would take some time to duplicate that setup.
I've tried to bump jest to the latest version (25.3.0) but wasn't able to merge. From what you said I understand that it should work if I align the libraries correctly and I see that @jest/reporters@25.3.0 is using istanbul-lib-coverage@3.0.0 so if I align that with my cypress instrumentation, it should work.
I'll try to make it work with jest@25.3.0, if the problems persists I'll make a demo project or else I'll close the issue.
EDIT I've tried with the following dependencies:
npmPackages:
istanbul-instrumenter-loader: 3.0.1 => 3.0.1
istanbul-lib-coverage: 3.0.0 => 3.0.0
nyc: ^15.0.1 => 15.0.1
source-map-support: ^0.5.16 => 0.5.16
ts-node: ^7.0.1 => 7.0.1
typescript: ~3.7.4 => 3.7.5
And jest@25.3.0. This resulted in Invalid file coverage object, missing keys, found:data
. Jest's coverage-final.json indeed contains data
properties. Does this relate to #1226?
I've replaced istanbul-instrumenter-loader with babel-plugin-istanbul, so the outdated istanbul-lib-instrument is gone. The coverage files still cannot be merged with the one Jest produces.
@D0rmouse the reason you're still getting this error Invalid file coverage object, missing keys, found:data
is because you aren't producing and merging the correct reporter file.
Make sure these happen:
json
reporter. If you're using jest it's something like this for its config:
"coverageReporters": [
"lcov",
"text",
"json"
]
The important part being that there's a json
entry there in the array
Then after you run the tests you'll have a coverage-final.json
created in the coverage dir. That's the file you need to copy to the combined directory along with the other file of the same name from a different directory. So both of these files should exist in a directory to combine them (give them different names)
Then run the nyc merge and report commands
Hope it helps 🤞 Goodluck!
@lirantal I do have a json entry in my coverageReporters array. I got json from the start it's just not merging correctly because the formats don't match.
@D0rmouse can you confirm that you have that file called coverage-final.json
created for both of your coverage report runs? Copy them to a directory, and then run the merge on that when they are both there alone with no other files.
@lirantal Yes I can, the output of the Jest test is simply different than my Cypress coverage output. I created an issue in the Jest repo as well. Can't reproduce using a minimal setup unfortunately. https://github.com/facebook/jest/issues/9808
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
I'm experiencing this same problem.
My coverage results look like so:
cypress/coverage-final.json
:
{
"path/to/file.ts": {
"path": "path/to/file.ts",
"statementMap": {
"0": {
"start": { "line": 4, "column": 35 },
"end": { "line": 4, "column": 66 }
},
"1": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": 3 }
},
"2": {
"start": { "line": 6, "column": 4 },
"end": { "line": 6, "column": 29 }
},
"3": {
"start": { "line": 8, "column": 2 },
"end": { "line": 8, "column": 14 }
},
"4": {
"start": { "line": 1, "column": 0 },
"end": { "line": 1, "column": 65 }
},
"5": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
},
"6": {
"start": { "line": 3, "column": 0 },
"end": { "line": 3, "column": 24 }
}
},
"fnMap": {
"0": {
"name": "getRoot",
"decl": {
"start": { "line": 3, "column": 24 },
"end": { "line": 3, "column": 31 }
},
"loc": {
"start": { "line": 3, "column": 47 },
"end": { "line": 9, "column": 1 }
},
"line": 3
},
"1": {
"name": "getRoot",
"decl": {
"start": { "line": 3, "column": 24 },
"end": { "line": 3, "column": 31 }
},
"loc": {
"start": { "line": 3, "column": 31 },
"end": { "line": 9, "column": 1 }
}
}
},
"branchMap": {
"0": {
"loc": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": 3 }
},
"type": "if",
"locations": [
{
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": 3 }
},
{ "start": {}, "end": {} }
],
"line": 5
},
"1": {
"loc": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
},
"type": "if",
"locations": [
{
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
}
]
}
},
"s": { "0": 6, "1": 4, "2": 2, "3": 4, "4": 2, "5": 2, "6": 2 },
"f": { "0": 4, "1": 2 },
"b": { "0": [0, 4], "1": [2] },
"_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
"hash": "81128de9166cac88dff871fa724f973dcd80167a"
}
}
jest/coverage-final.json
:
{
"path/to/file.ts": {
"path": "path/to/file.ts",
"statementMap": {
"0": {
"start": { "line": 1, "column": 0 },
"end": { "line": 1, "column": 65 }
},
"1": {
"start": { "line": 4, "column": 35 },
"end": { "line": 4, "column": 66 }
},
"2": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
},
"3": {
"start": { "line": 6, "column": 4 },
"end": { "line": 6, "column": 29 }
},
"4": {
"start": { "line": 8, "column": 2 },
"end": { "line": 8, "column": 14 }
},
"5": {
"start": { "line": 3, "column": 0 },
"end": { "line": 3, "column": 24 }
}
},
"fnMap": {
"0": {
"name": "getRoot",
"decl": {
"start": { "line": 3, "column": 24 },
"end": { "line": 3, "column": 31 }
},
"loc": {
"start": { "line": 3, "column": 31 },
"end": { "line": 9, "column": 1 }
}
}
},
"branchMap": {
"0": {
"loc": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
},
"type": "if",
"locations": [
{
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
}
]
}
},
"s": { "0": 1, "1": 1, "2": 1, "3": 1, "4": 0, "5": 1 },
"f": { "0": 1 },
"b": { "0": [1] }
}
}
The expected outcome would be path/to/file.ts
having 100% coverage, since the gaps on one test are covered by the other test. However, instead, what you see is the gaps being reported:
combined/coverage-final.json
:
{
"path/to/file.ts": {
"path": "path/to/file.ts",
"statementMap": {
"0": {
"start": { "line": 4, "column": 35 },
"end": { "line": 4, "column": 66 }
},
"1": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": 3 }
},
"2": {
"start": { "line": 6, "column": 4 },
"end": { "line": 6, "column": 29 }
},
"3": {
"start": { "line": 8, "column": 2 },
"end": { "line": 8, "column": 14 }
},
"4": {
"start": { "line": 1, "column": 0 },
"end": { "line": 1, "column": 65 }
},
"5": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
},
"6": {
"start": { "line": 3, "column": 0 },
"end": { "line": 3, "column": 24 }
}
},
"fnMap": {
"0": {
"name": "getRoot",
"decl": {
"start": { "line": 3, "column": 24 },
"end": { "line": 3, "column": 31 }
},
"loc": {
"start": { "line": 3, "column": 47 },
"end": { "line": 9, "column": 1 }
},
"line": 3
},
"1": {
"name": "getRoot",
"decl": {
"start": { "line": 3, "column": 24 },
"end": { "line": 3, "column": 31 }
},
"loc": {
"start": { "line": 3, "column": 31 },
"end": { "line": 9, "column": 1 }
}
}
},
"branchMap": {
"0": {
"loc": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": 3 }
},
"type": "if",
"locations": [
{
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": 3 }
},
{ "start": {}, "end": {} }
],
"line": 5
},
"1": {
"loc": {
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
},
"type": "if",
"locations": [
{
"start": { "line": 5, "column": 2 },
"end": { "line": 7, "column": null }
}
]
}
},
"s": { "0": 7, "1": 4, "2": 3, "3": 4, "4": 3, "5": 3, "6": 3 },
"f": { "0": 4, "1": 3 },
"b": { "0": [0, 4], "1": [3] },
"_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
"hash": "81128de9166cac88dff871fa724f973dcd80167a"
}
}
The "coverage" merge is really just concatenated instead of merging.
It appears that: line 1 characters 1-10 uncovered
+ line 1 characters 1-5 covered
+ line 1 characters 5-10 covered
= line 1 characters 1-10 uncovered
(wrong) + line 1 characters 1-10 covered
(right).
The uncovered characters are still shown as uncovered even though their character-span is covered by a different part of the map. Just guessing.
Here is a nice visual LCOV report for the above JSON:
Cypress (fully covered except for 1 branch):
Jest (specifically covering the above branch):
Combined report (expected to be fully covered):
Any advice to unblock this? I don't want to have to write a JSON merger manually. 😓
Maybe this is just a weird issue with TypeScript.
Below, how are imports and exports not covered? The export
statement isn't covered, but the inside of the function somehow is. The inside references an import, but the import isn't covered. Nonsensical. 😵
I am facing the same issue as the above user @CharlesStover Any update on how to fix this issue?
My issue ended up simply being that the version of Jest needed to be downgraded to 26. No idea why 27 is broken. The instrumentation of Jest 27 is simply off and doesn't match the one generated by react-scripts
when running the local application in dev mode for Cypress.
I am trying to integrate this to existing application which is using Jest 27.0.6 and can't downgrade to lower version. Any other way to fix this?
I spent a long time trying to fix it, and was not able to solve it with Jest 27. A downgrade to 26 resolved it like magic. This may need to be a bug report for the Jest team, as it appears 27 is not instrumenting coverage correctly, producing incorrect results for TypeScript files (this occurs with either ts-jest
or Babel for transpilation). Things like "import type" should not be showing as uncovered, ever.
If you want to open a ticket with the Jest team, I think that's the right solution, and you could link to my screenshots and data in this issue as a reference for them.
I have the same issue just by merging multiple reports from various cypress runs in parallel. The merge is just taking the last report base on the order:
coverage-final-A.json
{
"global.component.ts": {
// ...same value in both files
"s": {
"0": 0,
"1": 0,
"2": 0,
"3": 0,
"4": 0,
"5": 0,
"6": 0,
"7": 0,
"8": 0,
"9": 0,
"10": 0,
"11": 0,
"12": 0,
"13": 0,
"14": 0,
"15": 0,
"16": 0,
"17": 0,
"18": 0,
"19": 0,
"20": 64,
"21": 0
},
"f": { "0": 0 },
"b": { "0": [0, 0] }
}
}
coverage-final-B.json
{
"global.component.ts": {
// ...same value in both files
"s": {
"0": 77,
"1": 77,
"2": 5151,
"3": 77,
"4": 77,
"5": 5151,
"6": 77,
"7": 77,
"8": 5151,
"9": 77,
"10": 77,
"11": 5151,
"12": 77,
"13": 77,
"14": 5151,
"15": 77,
"16": 77,
"17": 77,
"18": 77,
"19": 77,
"20": 154,
"21": 77
},
"f": { "0": 77 },
"b": { "0": [77, 77] }
}
In this case, only the result of coverage-final-A.json
will be taken, so without coverage.
If I rename coverage-final-A.json
to coverage-final-Z.json
, then the order is different and the result of coverage-final-B.json
will be taken.
Any ideas? Should rewrite a merge myself?
@D0rmouse did you finally find a good configuration to combine multiple coverage reports?
I have the same type of stack with Nx + Cypress + Storybook + Jest, generating reports separately works fine but when I want to merge, it is not good.
It makes sense because each tool are using different versions for each generation.
Hi all,
Maybe it will help someone:
I had the same problem (merging of cypress and jest coverage wasn't right) and I got that fixed after adding coverageProvider into jest config:
coverageProvider: 'v8',
Same issue. Any updates. Screenshots below:
Cypress Coverage - correct:
Jest Coverage - correct:
Combined Coverage - wrong:
I'm using nyc merge and nyc report just like the original reporter of this issue. The final JSON just doesn't appear correct.
@mmisty I tried your suggestion and it didn't work. I end up with this instead:
@sscaff1, hi Steven After sometime I ended up with this cypress-template.
Key points as I remember:
node ./.scripts/merge.js --cypress reports/coverage-cypress --jest reports/coverage-jest --out reports/coverage-temp --report reports/coverage-full
In my case, I fixed this issue by using the babel coverage provider in my Jest config coverageProvider: 'babel',
. In my babel config I added a enviroment variable to check if the Code should be instrumented or not.
const shouldInstrumentCode = 'INSTRUMENT_CODE' in process.env
//console.log('shouldInstrumentCode', shouldInstrumentCode)
module.exports = {
presets: ["next/babel"],
plugins: shouldInstrumentCode? ["istanbul"]: []
};
To run this in instrument mode, add a script like this to your package.json "dev:test": "set INSTRUMENT_CODE=1 && next dev",
This finally provides accurate coverage results for me.
I was facing the same issue as OP mentioned, as well as of other people here in the thread.
My issue was a wrong merged report based on:
ts-jest
Snooping around at the code/produced reports, I found that:
instambul-lib-coverage
was not made to handle reports that disagreed on what lines/statements/functions/branches did or did not existSo the actual problem was getting one or the other inline with their counterpart.
First I tried to force my npm dependencies to all be the same, assuming this was the issue (different versions of istanbul, which I did have installed):
Example package.json
:
"overrides": {
"@cypress/code-coverage": {
"istanbul-lib-coverage": "3.2.0"
},
"nyc": {
"istanbul-lib-instrument": "5.2.1"
}
But that was a dud, as the problem lied with ts-jest
transformation strategy, and swapped that out and now I am using babel:
module.exports = {
presets: [['@babel/preset-env', { targets: { node: 'current' } }], ['@babel/preset-react', { runtime: 'automatic' }], '@babel/preset-typescript'],
}
That seems to have finally aligned the stars, and there a difference of about 300 lines, which I suspect is some other config I still have to tweak.
I'm encountering the same problem. When I try to merge two coverage-final.json files, the output is erroneous.
I have posted a Stack Overflow question with the details here .
I am having the same issue. I add both coverage files I generated:
This one was generated using vitest coverage-final.json This one was generated using the storybook addon coverage-storybook.json
Here are my configs for vitest
coverage: {
all: true,
clean: false,
exclude: [
...
],
provider: 'istanbul',
reporter: ['json', 'html']
}
},
// my command
"test:coverage": "vitest --run --coverage",
For storybook I used:
addons: [
{
name: '@storybook/addon-coverage',
options: {
istanbul: {
exclude: [
....
]
}
}
},
]
"test-storybook:coverage": "test-storybook --coverage --json --coverageDirectory=./coverage",
At the end I use nyc
to merge the output files:
"test-all:coverage": "nyc merge coverage coverage/merged/coverage.json && nyc report -t coverage/merged --report-dir coverage/merged --reporter=html --reporter=cobertura",
In my merged coverage report I can then see:
However, when looking at only the unit test report, I can see that these lines are being hitted.
Okay I digged a little bit and it seems that the storybook addon and vitest generate slightly different line and column for the statements.
For example in one component vitest said that there is a stmt on line 38 start at col 18 and ends on the same line.
"2": {
"start": {
"line": 38,
"column": 18
},
"end": {
"line": 38,
"column": null
}
},
However, storybook coverage report says the statement is on line 38 in column 2:
"3": {
"start": {
"line": 38,
"column": 2
},
"end": {
"line": 38,
"column": null
}
},
So now the question becomes how can I make sure that both instrumentalize the code the same way?
This happens with vitest 1.2.1, cypress 13.2.0, and @cypress/code-coverage 3.12.19 as well
I had the same problem, and when I merged the Cypress and Jest test reports, the result was obviously wrong.
Is there any solution?
I execute the following nyc commands, after generating individual coverage reports on the same codebase using different test executions run by Jest and Cypress.
nyc merge coverage-jest combined/coverage-jest.json
nyc merge coverage-cypress combined/coverage-cypress.json
nyc report combined --reporter html --reporter text
Expected Behavior
I'd expect this to show a correct cumulative coverage report of .ts and .html files.
Observed Behavior
Cannot read property 'locations' of undefined
TypeError
is shown.Troubleshooting steps
cache: false
in my nyc configEnvironment Information
For Cypress coverage I use above dependencies as well as
@cypress/code-coverage:^3.0.2
. I'm usingjest: 24.1.0
to produce unit test coverage.