check-coverage doesn’t work correctly

lo1tuma commented 8 years ago

I’m using istanbul 1.0.0-alpha2 with a setup similar to sample-babel-node. My npm test script looks like this:

NODE_ENV=test babel-node node_modules/istanbul/lib/cli.js cover _mocha test/unit -- --opts test/unit/mocha.opts && babel-node node_modules/istanbul/lib/cli.js check-coverage

If I run this script I get different results for the cover and check-coverage command:

cover command:

=============================== Coverage summary ===============================
Statements   : 100% ( 3802/3802 )
Branches     : 99.89% ( 945/946 )
Functions    : 100% ( 691/691 )
Lines        : 100% ( 3769/3769 )
================================================================================

check-coverage command:

Error: ERROR: Coverage for statements (90.41%) does not meet global threshold (100%)
ERROR: Coverage for branches (63.77%) does not meet global threshold (100%)
ERROR: Coverage for functions (99.27%) does not meet global threshold (100%)

As you can see the coverage for e.g. branches is one time 99.89% and the other time it is only 63.77%. I think the problem is that the check-coverage command reads the coverage result from coverage.raw.json which is the coverageMap before the source map was applied. I’ve changed this code on my local istanbul installation to write the content of transformed.map instead of finalCoverage to the coverage.raw.json file. This solved the problem for me.

gotwarlost commented 8 years ago

Could you try the new version I just pushed. check-coverage was erroneously computing coverage from the raw (pre-source map) files as opposed to the post processed files.

Reopen if the problem still exists.

lo1tuma commented 8 years ago

Hi @gotwarlost, I’ve tried istanbul-api version 1.0.0-alpha.10, but the problem still exists:

$ npm ls | grep istanbul
├─┬ istanbul@1.0.0-alpha.2
├─┬ istanbul-api@1.0.0-alpha.10
│ ├── istanbul-lib-coverage@1.0.0-alpha.4
│ ├─┬ istanbul-lib-hook@1.0.0-alpha.4
│ ├─┬ istanbul-lib-instrument@1.0.0-alpha.6
│ ├─┬ istanbul-lib-report@1.0.0-alpha.3
│ ├── istanbul-lib-source-maps@1.0.0-alpha.9
│ ├── istanbul-reports@1.0.0-alpha.4

Coverage summary:

=============================== Coverage summary ===============================
Statements   : 100% ( 207/207 )
Branches     : 100% ( 50/50 )
Functions    : 100% ( 81/81 )
Lines        : 100% ( 207/207 )
================================================================================

check-coverage output:

> istanbul check-coverage --statement 100 --branch 100 --function 100 --lines 100

/Users/foo/workspace/myproject/node_modules/istanbul/lib/cli.js:38
        throw ex; // turn it into an uncaught exception
        ^
ERROR: Coverage for statements (98.04%) does not meet global threshold (100%)
ERROR: Coverage for branches (85.85%) does not meet global threshold (100%)

gotwarlost commented 8 years ago

Dammit, will take a look. Thanks for testing. If this project is public on github, could you give me a link. Otherwise, I'll deal with reproducing it.

lo1tuma commented 8 years ago

Unfortunately the project is not opensource, but I’ve create a reproducible example based on sample-mocha-compilers.

lo1tuma commented 8 years ago

FWIW this change solved the issue for me, but I'm not sure if this is the correct way to solve the problem.

gotwarlost commented 8 years ago

Thanks for the repro case. I will look into in shortly.