search

cenfun / monocart-coverage-reports

A code coverage tool to generate native V8 reports or Istanbul reports.
MIT License
31 stars 5 forks source link
code code-coverage coverage istanbul monocart reporter reports testing v8

readme

Monocart Coverage Reports

🌐 English | 简体中文

A JavaScript code coverage tool to generate native V8 reports or Istanbul reports.

Usage

It's recommended to use Node.js 20+.

  • Install 
    npm install monocart-coverage-reports
  • API 
    const MCR = require('monocart-coverage-reports');
const mcr = MCR({
name: 'My Coverage Report - 2024-02-28',
outputDir: './coverage-reports',
reports: ["v8", "console-details"],
cleanCache: true
});
await mcr.add(coverageData);
await mcr.generate();

    Using import and load options from config file

    import { CoverageReport } from 'monocart-coverage-reports';
const mcr = new CoverageReport();
await mcr.loadConfig();

    For more information, see Multiprocessing Support

Options

Available Reports

V8 build-in reports (V8 data only):

Istanbul build-in reports (both V8 and Istanbul data):

Other build-in reports (both V8 and Istanbul data):

Multiple Reports:

const MCR = require('monocart-coverage-reports');
const coverageOptions = {
    outputDir: './coverage-reports',
    reports: [
        // build-in reports
        ['console-summary'],
        ['v8'],
        ['html', {
            subdir: 'istanbul'
        }],
        ['json', {
            file: 'my-json-file.json'
        }],
        'lcovonly',

        // custom reports
        // Specify reporter name with the NPM package
        ["custom-reporter-1"],
        ["custom-reporter-2", {
            type: "istanbul",
            key: "value"
        }],
        // Specify reporter name with local path
        ['/absolute/path/to/custom-reporter.js']
    ]
}
const mcr = MCR(coverageOptions);

Compare Reports

If the V8 data format is used for Istanbul reports, it will be automatically converted from V8 to Istanbul.

Istanbul V8 V8 to Istanbul
Coverage data Istanbul (Object) V8 (Array) V8 (Array)
Output Istanbul reports V8 reports Istanbul reports
- Bytes ❌ ✅ ❌
- Statements ✅ ✅ ✅
- Branches ✅ ✅ ✅
- Functions ✅ ✅ ✅
- Lines ✅ ✅ ✅
- Execution counts ✅ ✅ ✅
CSS coverage ❌ ✅ ✅
Minified code ❌ ✅ ❌

Collecting Istanbul Coverage Data

Collecting V8 Coverage Data

Collecting V8 Coverage Data with Playwright

await Promise.all([
    page.coverage.startJSCoverage({
        // reportAnonymousScripts: true,
        resetOnNavigation: false
    }),
    page.coverage.startCSSCoverage({
        // Note, anonymous styles (without sourceURLs) are not supported, alternatively, you can use CDPClient
        resetOnNavigation: false
    })
]);

await page.goto("your page url");

const [jsCoverage, cssCoverage] = await Promise.all([
    page.coverage.stopJSCoverage(),
    page.coverage.stopCSSCoverage()
]);

const coverageData = [... jsCoverage, ... cssCoverage];

Collect coverage with @playwright/test Automatic fixtures, see example: fixtures.ts For more examples, see ./test/test-v8.js, and anonymous, css

Collecting Raw V8 Coverage Data with Puppeteer

await Promise.all([
    page.coverage.startJSCoverage({
        // reportAnonymousScripts: true,
        resetOnNavigation: false,
        // provide raw v8 coverage data
        includeRawScriptCoverage: true
    }),
    page.coverage.startCSSCoverage({
        resetOnNavigation: false
    })
]);

await page.goto("your page url");

const [jsCoverage, cssCoverage] = await Promise.all([
    page.coverage.stopJSCoverage(),
    page.coverage.stopCSSCoverage()
]);

// to raw V8 script coverage
const coverageData = [... jsCoverage.map((it) => {
    return {
        source: it.text,
        ... it.rawScriptCoverage
    };
}), ... cssCoverage];

Example: ./test/test-puppeteer.js

Collecting V8 Coverage Data from Node.js

Possible solutions:

Collecting V8 Coverage Data with CDPClient API

startCSSCoverage: () => Promise; stopCSSCoverage: () => Promise<V8CoverageEntry[]>;

/* start both js and css coverage / startCoverage: () => Promise; /* stop and return both js and css coverage / stopCoverage: () => Promise<V8CoverageEntry[]>;

/* write the coverage started by NODE_V8_COVERAGE to disk on demand, returns v8 coverage dir / writeCoverage: () => Promise;

/* get istanbul coverage data / getIstanbulCoverage: (coverageKey?: string) => Promise;


- Work with node debugger port `--inspect=9229` or browser debugging port `--remote-debugging-port=9229`
```js
const MCR = require('monocart-coverage-reports');
const client = await MCR.CDPClient({
    port: 9229
});
await client.startJSCoverage();
// run your test here
const coverageData = await client.stopJSCoverage();

V8 Coverage Data API

Filtering Results

Using entryFilter and sourceFilter to filter the results for V8 report

When V8 coverage data collected, it actually contains the data of all entry files, for example:

We can use entryFilter to filter the entry files. For example, we should remove vendor.js and something-else.js if they are not in our coverage scope.

When inline or linked sourcemap exists to the entry file, the source files will be extracted from the sourcemap for the entry file, and the entry file will be removed if logging is not debug.

We can use sourceFilter to filter the source files. For example, we should remove dependency.js if it is not in our coverage scope.

For example:

const coverageOptions = {
    entryFilter: (entry) => entry.url.indexOf("main.js") !== -1,
    sourceFilter: (sourcePath) => sourcePath.search(/src\//) !== -1
};

Or using minimatch pattern:

const coverageOptions = {
    entryFilter: "**/main.js",
    sourceFilter: "**/src/**"
};

Support multiple patterns:

const coverageOptions = {
    entryFilter: {
        '**/node_modules/**': false,
        '**/vendor.js': false,
        '**/src/**': true
    },
    sourceFilter: {
        '**/node_modules/**': false,
        '**/**': true
    }
};

As CLI args (JSON-like string. Added in: v2.8):

mcr --sourceFilter "{'**/node_modules/**':false,'**/**':true}"

Note, those patterns will be transformed to a function, and the order of the patterns will impact the results:

const coverageOptions = {
    entryFilter: (entry) => {
        if (minimatch(entry.url, '**/node_modules/**')) { return false; }
        if (minimatch(entry.url, '**/vendor.js')) { return false; }
        if (minimatch(entry.url, '**/src/**')) { return true; }
        return false; // else unmatched
    }
};

Using filter instead of entryFilter and sourceFilter

If you don't want to define both entryFilter and sourceFilter, you can use filter instead. (Added in: v2.8)

const coverageOptions = {
    // combined patterns
    filter: {
        '**/node_modules/**': false,
        '**/vendor.js': false,
        '**/src/**': true
        '**/**': true
    }
};

Resolve sourcePath for the Source Files

If the source file comes from the sourcemap, then its path is a virtual path. Using the sourcePath option to resolve a custom path. For example, we have tested multiple dist files, which contain some common files. We hope to merge the coverage of the same files, so we need to unify the sourcePath in order to be able to merge the coverage data.

const coverageOptions = {
    sourcePath: (filePath) => {
        // Remove the virtual prefix
        const list = ['my-dist-file1/', 'my-dist-file2/'];
        for (const str of list) {
            if (filePath.startsWith(str)) {
                return filePath.slice(str.length);
            }
        }
        return filePath;
    }
};

It also supports simple key/value replacement:

const coverageOptions = {
    sourcePath: {
        'my-dist-file1/': '', 
        'my-dist-file2/': ''
    }
};

Normalize the full path of the file:

const path = require("path")

// MCR coverage options
const coverageOptions = {
    sourcePath: (filePath, info)=> {
        if (!filePath.includes('/') && info.distFile) {
            return `${path.dirname(info.distFile)}/${filePath}`;
        }
        return filePath;
    }
}

Adding Empty Coverage for Untested Files

By default the untested files will not be included in the coverage report, we can add empty coverage data for all files with option all, the untested files will show 0% coverage.

const coverageOptions = {
    all: {
        dir: ['./src'],
        filter: (filePath) => {
            return true;
        }
    }
};

The filter also supports minimatch pattern:

const coverageOptions = {
    all: {
        dir: ['./src'],
        filter: '**/*.js'
    }
};
// or multiple patterns
const coverageOptions = {
    all: {
        dir: ['./src'],
        filter: {
            // exclude files
            '**/ignored-*.js': false,
            '**/*.html': false,
            '**/*.ts': false,
            // empty css coverage
            '**/*.scss': "css",
            '**/*': true
        }
    }
};

onEnd Hook

For example, checking thresholds:

const EC = require('eight-colors');
const coverageOptions = {
    name: 'My Coverage Report',
    outputDir: './coverage-reports',
    onEnd: (coverageResults) => {
        const thresholds = {
            bytes: 80,
            lines: 60
        };
        console.log('check thresholds ...', thresholds);
        const errors = [];
        const { summary } = coverageResults;
        Object.keys(thresholds).forEach((k) => {
            const pct = summary[k].pct;
            if (pct < thresholds[k]) {
                errors.push(`Coverage threshold for ${k} (${pct} %) not met: ${thresholds[k]} %`);
            }
        });
        if (errors.length) {
            const errMsg = errors.join('\n');
            console.log(EC.red(errMsg));
            // throw new Error(errMsg);
            // process.exit(1);
        }
    }
}

Ignoring Uncovered Codes

To ignore codes, use the special comment which starts with v8 ignore:

const os = platform === 'wind32' ? 'Windows' / v8 ignore next / : 'Other';

// v8 ignore next 3 if (platform === 'linux') { console.log('hello linux'); }


## Multiprocessing Support
> The data will be added to `[outputDir]/.cache`, After the generation of the report, this data will be removed unless debugging has been enabled or a raw report has been used, see [Debug for Coverage and Sourcemap](#debug-for-coverage-and-sourcemap)
- Main process, before the start of testing
```js
const MCR = require('monocart-coverage-reports');
const coverageOptions = require('path-to/same-options.js');
const mcr = MCR(coverageOptions);
// clean previous cache before the start of testing
// unless the running environment is new and no cache
mcr.cleanCache();

Command Line

The CLI will run the program as a child process with NODE_V8_COVERAGE=dir until it exits gracefully, and generate the coverage report with the coverage data from the dir.

Config File

Loading config file by priority:

Merge Coverage Reports

The following usage scenarios may require merging coverage reports:

Automatic Merging

Manual Merging

If the reports cannot be merged automatically, then here is how to manually merge the reports.
First, using the raw report to export the original coverage data to the specified directory.

Common issues

Unexpected coverage

In most cases, it happens when the coverage of the generated code is converted to the coverage of the original code through a sourcemap. In other words, it's an issue with the sourcemap. Most of the time, we can solve this by setting minify to false in the configuration of build tools. Let's take a look at an example:

const a = tf ? 'true' : 'false';
               ^     ^  ^
              m1     p  m2

In the generated code, there is a position p, and we need to find out its corresponding position in the original code. Unfortunately, there is no matched mapping for the position p. Instead, it has two adjacent upstream and downstream mappings m1 and m2, so, the original position of p that we are looking for, might not be able to be precisely located. Especially, the generated code is different from the original code, such as the code was minified, compressed or converted, it is difficult to find the exact original position without matched mapping.

How MCR Works:

Unparsable source

It happens during the parsing of the source code into AST, if the source code is not in the standard ECMAScript. For example ts, jsx and so on. There is a option to fix it, which is to manually compile the source code for these files.

import * as fs from "fs";
import * as path from "path";
import { fileURLToPath } from "url";
import * as TsNode from 'ts-node';
const coverageOptions = {
    onEntry: async (entry) => {
        const filePath = fileURLToPath(entry.url)
        const originalSource = fs.readFileSync(filePath).toString("utf-8");
        const fileName = path.basename(filePath);
        const tn = TsNode.create({});
        const source = tn.compile(originalSource, fileName);
        entry.fake = false;
        entry.source = source;
    }
}

JavaScript heap out of memory

When there are a lot of raw v8 coverage files to process, it may cause OOM. We can try the following Node.js options:

- run: npm run test:coverage
    env:
        NODE_OPTIONS: --max-old-space-size=8192

Debug for Coverage and Sourcemap

Sometimes, the coverage is not what we expect. The next step is to figure out why, and we can easily find out the answer step by step through debugging.

  • Start debugging for v8 report with option logging: 'debug' 
    const coverageOptions = {
logging: 'debug',
reports: [
['v8'],
['console-details']
]
};

    When logging is debug, the raw report data will be preserved in [outputDir]/.cache or [outputDir]/raw if raw report is used. And the dist file will be preserved in the V8 list, and by opening the browser's devtool, it makes data verification visualization effortless.

Integration with Any Testing Framework

Integration Examples

Playwright

c8

CodeceptJS

VSCode

Jest

Vitest

Node Test Runner

Puppeteer

Cypress

WebdriverIO

Storybook Test Runner

TestCafe

Selenium Webdriver

Mocha

mcr mocha ./test/**/*.js
mcr --import tsx mocha ./test/**/*.ts

tsx

mcr --import tsx tsx ./src/example.ts

ts-node

AVA

mcr ava

Codecov

codecov

Codacy

Codacy

Coveralls

Coverage Status

Sonar Cloud

Coverage

Contributing

npm run build npm run test

npm run dev

- Refreshing `eol=lf` for snapshot of test (Windows)
```sh
git add . -u
git commit -m "Saving files before refreshing line endings"

npm run eol

Thanks