Add One Double Zero as coverage provider

ericmorand commented 3 weeks ago

Summary

As explained in the issue, v8 coverage provider comes with some important tradeoffs compared to the babel/istanbul one. One Double Zero is a code coverage tool and API that consumes V8 coverage data and targets the accuracy and correctness of istanbul. It does this by operating at the AST level.

This PR adds One Double Zero as a coverage provider.

It also updates the documentation, and explains the tradeoffs of the v8 coverage provider.

Test plan

The plan is to execute odz on each e2e test executed by the v8 coverage provider test suite, and use the output result as the snapshot for e2e/__tests__/coverageProviderODZ.test.ts. The test script of each e2e has to be changed to be executable wth either node or ts-node, which does not impact the coverage result.

e2e/coverage-provider-v8/cjs-native-without-sourcemap

e2e/coverage-provider-v8/cjs-native-without-sourcemap$ odz --sources=module.js --sources=uncovered.js node __tests__/test.js

this will print
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |      60 |       50 |      50 |      60 |                   
 module.js    |   66.66 |       50 |      50 |   66.66 | 14-15,19          
 uncovered.js |       0 |      100 |     100 |       0 | 8                 
--------------|---------|----------|---------|---------|-------------------

e2e/coverage-provider-v8/cjs-with-babel-transformer

e2e/coverage-provider-v8/cjs-with-babel-transformer$ odz --sources=module.ts --sources=uncovered.ts --sources=types.ts ts-node -T __tests__/test.ts

this will print
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |    62.5 |       50 |      50 |    62.5 |                   
 module.ts    |    62.5 |       50 |      50 |    62.5 | 16-17,21          
 types.ts     |       0 |        0 |       0 |       0 |                   
 uncovered.ts |       0 |        0 |       0 |       0 |                   
--------------|---------|----------|---------|---------|-------------------

e2e/coverage-provider-v8/empty-sourcemap

e2e/coverage-provider-v8/empty-sourcemap$ odz  --sources=types.ts ts-node -T __tests__/test.ts

----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |       0 |        0 |       0 |       0 |                   
 types.ts |       0 |        0 |       0 |       0 |                   
----------|---------|----------|---------|---------|-------------------

e2e/coverage-provider-v8/esm-native-without-sourcemap

e2e/coverage-provider-v8/esm-native-without-sourcemap$ odz --sources=module.js node __tests__/test.js 

--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |    62.5 |       50 |      50 |    62.5 |                   
 module.js    |    62.5 |       50 |      50 |    62.5 | 14-15,19          
 uncovered.js |       0 |        0 |       0 |       0 |                   
--------------|---------|----------|---------|---------|-------------------

e2e/coverage-provider-v8/esm-with-custom-transformer

e2e/coverage-provider-v8/esm-with-custom-transformer$ odz --sources=module.ts --sources=uncovered.ts --sources=types.ts ts-node -T __tests__/test.ts

--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |    62.5 |       50 |      50 |    62.5 |                   
 module.ts    |    62.5 |       50 |      50 |    62.5 | 16-17,21          
 types.ts     |       0 |        0 |       0 |       0 |                   
 uncovered.ts |       0 |        0 |       0 |       0 |                   
--------------|---------|----------|---------|---------|-------------------

e2e/coverage-provider-v8/no-sourcemap

e2e/coverage-provider-v8/no-sourcemap$ odz --sources=Thing.js --sources=x.css node __tests__/Thing.test.js

42
----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |     100 |      100 |     100 |     100 |                   
 Thing.js |     100 |      100 |     100 |     100 |                   
 x.css    |       0 |        0 |       0 |       0 |                   
----------|---------|----------|---------|---------|-------------------

e2e/coverage-provider-v8/with-resetModules

e2e/coverage-provider-v8/with-resetModules$ odz --sources=module.js --sources=uncovered.js node __tests__/test.js

this will print
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |      60 |       50 |      50 |      60 |                   
 module.js    |   66.66 |       50 |      50 |   66.66 | 14-15,19          
 uncovered.js |       0 |      100 |     100 |       0 | 8                 
--------------|---------|----------|---------|---------|-------------------

e2e/vmscript-coverage

e2e/vmscript-coverage$ odz --sources=module.js --sources=package/vmscript.js node __tests__/extract-coverage.test.js

-------------|---------|----------|---------|---------|-------------------
File         | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-------------|---------|----------|---------|---------|-------------------
All files    |      80 |       75 |   66.66 |      80 |                   
 vmscript.js |      80 |       75 |   66.66 |      80 | 20-21             
-------------|---------|----------|---------|---------|-------------------

A few changes had to be made, like passing the list of files to cover to the _getCoverageResult method, but globally the changes required to add odz were minimal.

linux-foundation-easycla[bot] commented 3 weeks ago

The committers listed above are authorized under a signed CLA.

:white_check_mark: login: ericmorand / name: Eric MORAND (897321787272e301b8aecc0c30f66dddbf511de4)

netlify[bot] commented 3 weeks ago

Deploy Preview for jestjs ready!

Built without sensitive environment variables

Name	Link
Latest commit	897321787272e301b8aecc0c30f66dddbf511de4
Latest deploy log	https://app.netlify.com/sites/jestjs/deploys/671c114d079a7000082d6590
Deploy Preview	https://deploy-preview-15356--jestjs.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

ericmorand-sonarsource commented 3 weeks ago

Another interesting test is using jest to check the coverage of the jest project itself, with every istanbul ignore pragma removed for consistency.

Using babel as provider

$ yarn jest --coverage --coverageProvider=babel

------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
File                                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                                                                     
------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
All files                                 |   68.51 |    65.55 |   65.33 |   68.54 |                                                                                                                                       

Test Suites: 1 failed, 470 passed, 471 total
Tests:       1 failed, 51 skipped, 5132 passed, 5184 total
Snapshots:   1766 passed, 1766 total
Time:        96.649 s
Ran all test suites in 15 projects.

Using v8 as provider

$ yarn jest --coverage --coverageProvider=v8

------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
File                                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                                                                     
------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
All files                                 |   68.33 |    86.69 |   75.97 |   68.33 |                                                                                                                                       

Test Suites: 1 failed, 470 passed, 471 total
Tests:       1 failed, 51 skipped, 5132 passed, 5184 total
Snapshots:   1766 passed, 1766 total
Time:        81.74 s, estimated 90 s
Ran all test suites in 15 projects.

Using odz as provider

$ yarn jest --coverage --coverageProvider=odz

------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
File                                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                                                                     
------------------------------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------------------------------------------
All files                                 |   65.68 |    61.61 |   64.67 |   65.76 |                                                                                                                                    

Test Suites: 1 failed, 470 passed, 471 total
Tests:       1 failed, 51 skipped, 5132 passed, 5184 total
Snapshots:   1766 passed, 1766 total
Time:        76.922 s, estimated 79 s
Ran all test suites in 15 projects.

The most striking difference is the function and branch coverage reported by v8 that is way too high compared to the two others. This is expected and is a well-known issue:

the v8 provider doesn't consider the relevancy of lines: empty lines and comments are considered as covered.
the v8 provider considers that every single file contains at least one covered function.

Using v8 coverage data without knowledge of the meaning of each line of code makes for a very inaccurate coverage computation. Both babel and odz know about the meaning of each line of code: babel parses the code at instrumentation phase; odz parses the source files after the execution of the script.

The differences between babel and odz, minor, may come from how the tools work: babel parses and instruments the executed script; odz parses the source files. It is likely that both approaches lead to slightly different results. In any case, the ultimate goal of One Double Zero is to match istanbul accuracy, so the differences here are likely to become less and less important as One Double Zero continues to improve.

About performance differences: without a proper benchmark under some controlled environment and context, and multiple executions, they don't mean much. But it is likely that babel is slower in the end: instrumenting and running an instrumented code is more taxing than just executing the code and parsing the source files.

jestjs / jest