search

cenfun / monocart-reporter

A playwright test reporter (Node.js)
https://cenfun.github.io/monocart-reporter/
MIT License
207 stars 12 forks source link
chart comments coverage discord e2e email grid html jira monocart-reporter playwright slack summary tags test testrail xray zephyr

readme

Monocart Reporter

Preview

https://cenfun.github.io/monocart-reporter

(For Github actions, we can enforce color with env: FORCE_COLOR: true)

Install

npm i -D monocart-reporter

Playwright Config

Note: Most examples use CommonJS by default, please move to ESM according to your needs.

// playwright.config.js
module.exports = {
reporter: [
['list'],
['monocart-reporter', {  
name: "My Test Report",
outputFile: './monocart-report/index.html'
}]
]
};

Playwright Docs https://playwright.dev/docs/test-reporters

Examples

Output

Reporter Options

View Trace Online

The Trace Viewer requires that the trace file must be loaded over the http:// or https:// protocols without CORS issue.

  • Start a local web server with following CLI: 
    
# serve and open report
npx monocart show-report <path-to-report>

serve report

npx monocart serve-report 

The server add the http header `Access-Control-Allow-Origin: *` to [allow requesting from any origin](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin), it works with `http://localhost:port/` or `http://127.0.0.1:port/`
- To successfully work with other `IP` or `domain`, you can start web server with `https`:
```sh
npx monocart show-report <path-to-report> --ssl <path-to-key,path-to-cert>

For example: npx monocart show-report monocart-report/index.html --ssl ssl/key.pem,ssl/cert.pem

You can create and install local CA with mkcert

Custom Fields Report

You can add custom fields to the report. for example: Owner, JIRA Key etc.

Custom Columns

The report will be displayed in a Tree Grid. The columns function is used to customize the grid columns. The column properties following:

Searchable Fields

// playwright.config.js
module.exports = {
     reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './monocart-report/index.html',
            columns: (defaultColumns) => {
                const locationColumn = defaultColumns.find((column) => column.id === 'location');
                locationColumn.searchable = true;
            }
        }]
    ]
};

Custom Fields in Comments

The code comments are good enough to provide extra information without breaking existing code, and no dependencies, clean, easy to read, etc.

  • First, enable option customFieldsInComments to true 
    // playwright.config.js
module.exports = {
reporter: [
['monocart-reporter', {  
// enable/disable custom fields in comments. Defaults to true.
customFieldsInComments: true
}]
]
};

For example, adding owner and jira to the cases, steps, and suites. or updating the value if the field exists like title

/**
 * for file (comment file in the first line)
 * @owner FO
 */
const { test, expect } = require('@playwright/test');

/**
 * for case
 * @owner Kevin
 * @jira MCR-16888
 */
test('case title', () => { 

});

/**
 * @description multiple lines text description
multiple lines text description
multiple lines text description
 * @jira MCR-16888
*/
test('case description', () => {

});

/**
 * for describe suite
 * @owner Mark
 * @jira MCR-16900
 */
test.describe('suite title', () => {

    test('case title', ({ browserName }, testInfo) => {

        /**
         * rewrite assert step title "expect.toBe" to
         * @title my custom assert step title
         * @annotations important
         */
        expect(testInfo).toBe(test.info());

        // @owner Steve
        await test.step('step title', () => {

        });

    });

});

/**
 * rewrite "beforeAll hook" title to
 * @title do something before all
 */
test.beforeAll(() => { 

});

/**
 * rewrite "beforeEach hook" title to
 * @title do something before each
 */
test.beforeEach(() => { 

});

Create Diagrams and Visualizations with Mermaid

A[Hard] -->|Text| B(Round) B --> C{Decision} C -->|One| D[Result 1] C -->|Two| E[Result 2]

*/
test('case description', () => {

});

see Mermaid doc

Custom Fields with setMetadata()

Using comments is only applicable to statically created tests, while using API setMetadata() can be applicable to all situations, which is including dynamically created tests.

const { test } = require('@playwright/test');
const { setMetadata } = require('monocart-reporter');
test.describe('Data Driven Tests with setMetadata(data, testInfo)', () => {
    const list = [{
        title: 'Example Case 1 Data Driven Test',
        owner: 'Jensen',
        jira: 'MCR-16889',
    }, {
        title: 'Example Case 2 Data Driven Test',
        owner: 'Mark',
        jira: 'MCR-16899'
    }];
    list.forEach((item, i) => {
        test(item.title, () => {
            setMetadata({
                owner: item.owner,
                jira: item.jira
            }, test.info());

            //expect(1).toBe(1);

        });
    });
});

Custom Data Visitor

The visitor function will be executed for each row item (suite, case and step). Arguments:

Collect Data from the Title

For example, we want to parse out the jira key from the title:

test('[MCR-123] collect data from the title', () => {

});

You can simply use regular expressions to parse and get jira key:

// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './monocart-report/index.html',
            visitor: (data, metadata) => {
                // [MCR-123] collect data from the title
                const matchResult = metadata.title.match(/\[(.+)\]/);
                if (matchResult && matchResult[1]) {
                    data.jira = matchResult[1];
                }
            }
        }]
    ]
};

multiple matches example: collect-data

Collect Data from the Annotations

It should be easier than getting from title. see custom annotations via test.info().annotations

test('collect data from the annotations', () => {
    test.info().annotations.push({
        type: "jira",
        description: "MCR-123"
    })
});
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './monocart-report/index.html',
            visitor: (data, metadata) => {
                // collect data from the annotations
                if (metadata.annotations) {
                    const jiraItem = metadata.annotations.find((item) => item.type === 'jira');
                    if (jiraItem && jiraItem.description) {
                        data.jira = jiraItem.description;
                    }
                }
            }
        }]
    ]
};

Remove Secrets and Sensitive Data

The report may hosted outside of the organization’s internal boundaries, security becomes a big issue. Any secrets or sensitive data, such as usernames, passwords, tokens and API keys, should be handled with extreme care. The following example is removing the password and token from the report data with the string replacement in visitor function.

// playwright.config.js
module.exports = {
reporter: [
['monocart-reporter', {  
name: "My Test Report",
outputFile: './monocart-report/index.html',
visitor: (data, metadata) => {
const mySecrets = [process.env.PASSWORD, process.env.TOKEN];
mySecrets.forEach((secret) => {
// remove from title
data.title = data.title.replace(secret, '***');
// remove from logs
if (data.logs) {
data.logs = data.logs.map((item) => item.replace(secret, '***'));
}
// remove from errors
if (data.errors) {
data.errors = data.errors.map((item) => item.replace(secret, '***'));
}
});
}
}]
]
};

see example: remove-secrets

Style Tags

// new syntax for tag in playwright v1.42.0 test('test title', { tag: ['@smoke', '@critical'] }, () => { }); test.describe('describe title', { tag: ['@smoke', '@critical'] }, () => { });


* Custom tag style
```js
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './monocart-report/index.html',
            tags: {
                smoke: {
                    style: {
                        background: '#6F9913'
                    },
                    description: 'This is Smoke Test'
                },
                critical: {
                    background: '#c00'
                }
            }
        }]
    ]
};

Metadata

All metadata will be listed in the report in a key/value format.

  • Global level metadata 
    
// playwright.config.js
module.exports = {
globalSetup: require.resolve('./common/global-setup.js'),
metadata: {
product: 'Monocart',
env: 'STG',
type: 'Regression',
executor: 'Mono',
    // test home page object model
    url: 'https://www.npmjs.org/package/monocart-reporter'
},
 reporter: [
    ['monocart-reporter', {  
        name: "My Test Report",
        outputFile: './monocart-report/index.html'
    }]
]

};


- Project level `metadata`
```js
// playwright.config.js
module.exports = {
    projects: [
        {
            name: 'Desktop Chromium',
            use: {
                browserName: 'chromium'
            },
            metadata: {
                projectData: 'project level metadata',
                owner: 'PO',
                link: 'https://github.com/cenfun/monocart-reporter'
            }
        }
    ]
}

Trend Chart

Note: The trend chart requires historical data generally stored in the server database. There is a serverless solution which is connecting and collecting historical trend data from previous report data before test every time.

  • If a report is generated in the same place every time, you can simply connect the data with the report JSON path (the data is not 100% safe if there is any runtime error, the previous output dir will be empty by Playwright but the reporter processing not finish) 
    // playwright.config.js
module.exports = {
reporter: [
['monocart-reporter', {  
name: "My Test Report",
outputFile: './monocart-report/index.html',
// connect previous report data for trend chart
trend: './monocart-report/index.json'
}]
]
};
  • Recommended: resolve the data by yourself (could be requested from the server), required data fields:
  • date (Number) the previous test date in milliseconds
  • duration (Number) the previous test duration
  • summary (Object) the previous test summary
  • trends (Array) historical data list, but except the previous self 
    // playwright.config.js
module.exports = {
reporter: [
['monocart-reporter', {  
name: "My Test Report",
outputFile: './monocart-report/index.html',
// connect previous report data for trend chart
trend: async () => {
const previousReportData = await readDataFromSomeWhere("path-to/report.json");
// do some data filtering to previous trends
previousReportData.trends = previousReportData.trends.filter((item) => {
// remove data a week ago
return item.date > (Date.now() - 7 * 24 * 60 * 60 * 1000)
});
return previousReportData;
}
}]
]
};

Code Coverage Report

The reporter integrates monocart-coverage-reports for coverage reports, there are two APIs:

Global Coverage Report

The global coverage report will merge all coverages into one global report after all the tests are finished.

const test = testBase.extend({ autoTestFixture: [async ({ page }, use) => {

    // NOTE: it depends on your project name
    const isChromium = test.info().project.name === 'Desktop Chromium';

    // console.log('autoTestFixture setup...');
    // coverage API is chromium only
    if (isChromium) {
        await Promise.all([
            page.coverage.startJSCoverage({
                resetOnNavigation: false
            }),
            page.coverage.startCSSCoverage({
                resetOnNavigation: false
            })
        ]);
    }

    await use('autoTestFixture');

    // console.log('autoTestFixture teardown...');
    if (isChromium) {
        const [jsCoverage, cssCoverage] = await Promise.all([
            page.coverage.stopJSCoverage(),
            page.coverage.stopCSSCoverage()
        ]);
        const coverageList = [... jsCoverage, ... cssCoverage];
        // console.log(coverageList.map((item) => item.url));
        await addCoverageReport(coverageList, test.info());
    }

}, {
    scope: 'test',
    auto: true
}]

}); export { test, expect };

- Adding server side coverage on global teardown
> For example, a Node.js web server start at the beginning of the test with the env `NODE_V8_COVERAGE=dir`, the V8 coverage data will be saved to `dir` with calling API `v8.takeCoverage()` manually or terminating server gracefully. On global teardown, reading all from `dir` and adding them to global coverage report. For Node.js, the V8 coverage data requires appending source manually.
```js
// global-teardown.js
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import { addCoverageReport } from 'monocart-reporter';

export default async (config) => {

    const dir = "your-v8-coverage-data-dir";

    const files = fs.readdirSync(dir);
    for (const filename of files) {
        const content = fs.readFileSync(path.resolve(dir, filename)).toString('utf-8');
        const json = JSON.parse(content);
        let coverageList = json.result;
        coverageList = coverageList.filter((entry) => entry.url && entry.url.startsWith('file:'));

        // appending source
        coverageList.forEach((entry) => {
            entry.source = fs.readFileSync(fileURLToPath(entry.url)).toString('utf8');
        });

        // there is no test info on teardown, just mock one with required config
        const mockTestInfo = {
            config
        };
        await addCoverageReport(coverageList, mockTestInfo);
    }
}

see Collecting V8 Coverage Data from Node.js

Coverage Options

Coverage Examples

Attach Lighthouse Audit Report

Attach an audit report with API attachAuditReport(runnerResult, testInfo). Arguments:


## Attach Network Report
Attach a network report with API `attachNetworkReport(har, testInfo)`. Arguments:
- `har` HAR path (String) or HAR file buffer (Buffer). see [HAR 1.2 Spec](http://www.softwareishard.com/blog/har-12-spec/)
- `testInfo` see [TestInfo](https://playwright.dev/docs/api/class-testinfo)

 Generate HAR with `recordHar` option in browser.newContext() (see example: [report-network.spec.js](https://github.com/cenfun/monocart-reporter/blob/main/tests/report-network/report-network.spec.js) preview [report](https://cenfun.github.io/monocart-reporter/network-1a18723ee59b36867898/index.html))

```js
const fs = require('fs');
const path = require('path');
const { test } = require('@playwright/test');
const { attachNetworkReport } = require('monocart-reporter');
let context;
test.describe('attach network report 1', () => {

    const harPath = path.resolve('.temp/network-report1.har');
    if (fs.existsSync(harPath)) {
        // remove previous
        fs.rmSync(harPath);
    }

    test('first, open page', async ({ browser }) => {
        context = await browser.newContext({
            recordHar: {
                path: harPath
            }
        });
        const page = await context.newPage();
        await page.goto('https://github.com/cenfun/monocart-reporter');
    });

    test('next, run test cases', async () => {

    });

    test('finally, attach HAR', async () => {
        // Close context to ensure HAR is saved to disk.
        await context.close();
        await attachNetworkReport(harPath, test.info());
    });
});

Generate HAR with playwright-har

import { test } from '@playwright/test';
import { attachNetworkReport } from 'monocart-reporter';
import { PlaywrightHar } from 'playwright-har';

const harPath = path.resolve('.temp/network-report2.har');
if (fs.existsSync(harPath)) {
    // remove previous
    fs.rmSync(harPath);
}

test('first, open page', async ({ browser }) => {
    const context = await browser.newContext();
    const page = await context.newPage();
    playwrightHar = new PlaywrightHar(page);
    await playwrightHar.start();
    await page.goto('https://github.com/cenfun/monocart-reporter');
});

test('next, run test cases', async () => {

});

test('finally, attach HAR', async () => {
    await playwrightHar.stop(harPath);
    await attachNetworkReport(harPath, test.info());
});

Preview Network HTML Report

Global State Management

When tests are executed in isolation mode, the reporter and each test may run in a different process, they cannot share data with each other. we can start a local WebSocket server to serve the global data, and read/write the global data with useState API from a test.

Setup Global State

module.exports = {
    reporter: [
        ['list'],
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './monocart-report/index.html',
            state: {
                data: {
                    count: 0
                },
                server: {
                    // port: 8130
                },
                onClose: (data, config) => {
                    // save state data to global metadata
                    Object.assign(config.metadata, data);
                }
            }
        }]
    ]
};

Get, Set, and Remove Global Data

const { test } = require('@playwright/test');
const { useState } = require('monocart-reporter');
test('state test', async ({ browserName }) => {
    const state = useState({
        // port: 8130
    });

    const count = await state.get('count');
    console.log('count', count);

    await state.set('count', count + 1);

    await state.set({
        browser: browserName,
        someKey: 'some value'
    });

    const [browser, someKey] = await state.get('browser', 'someKey');
    console.log(browser, someKey);

    await state.remove('someKey');

    const all = await state.get();
    console.log(all);
});

Send and Receive Messages between Processes

Merge Shard Reports

There will be multiple reports to be generated if Playwright test executes in sharding mode. for example:

npx playwright test --shard=1/3
npx playwright test --shard=2/3
npx playwright test --shard=3/3

There are 3 reports will be generated.

Using merge API to merge all reports into one

Note: One more suite level "shard" will be added, its title will be the machine hostname, and the summary will be restated. All attachments will be copied to the merged output directory.


import { merge } from 'monocart-reporter';

// json file path const reportDataList = [ 'path-to/shard1/index.json', 'path-to/shard2/index.json', 'path-to/shard3/index.json'

// Or load zip file directly if the output files is zipped
// 'path-to/shard1/index.zip',
// 'path-to/shard2/index.zip',
// 'path-to/shard3/index.zip'

];

await merge(reportDataList, { name: 'My Merged Report', outputFile: 'merged-report/index.html', onEnd: async (reportData, helper) => { // send email or third party integration } });


> Note: The coverage reports will be merged automatically if we specify the `raw` report in coverage options:
```js
// global coverage options
coverage: {
    reports: [
        // for merging coverage reports
        'raw'
        // we can merge and zip the raw report files
        // ['raw', { merge: true, zip: true }]
    ]
}

see example merge.js

Using merge CLI

npx monocart merge <glob-patterns>

# -o --outputFile
npx monocart merge path-to/shard*/index.json -o merged-reports/index.html

# -c --config
npx monocart merge path-to/shard*/my-report.zip -c mr.config.js

# NOTE: The asterisk(*) is a special character which is interpreted by some operating systems
# For example: Mac and Linux, please put it in quotes, but NOT for Windows
npx monocart merge 'path-to/shard*/*.zip'

The default config files (In order of priority)

Preload for TypeScript config file:

onEnd Hook

The onEnd function will be executed after report generated. Arguments:

// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './monocart-report/index.html',
            // async hook after report data generated
            onEnd: async (reportData, helper) => {
                // console.log(reportData.summary);

                // find a test by title
                const myCase = helper.find((item, parent) => item.type === 'case' && item.title.includes('inline tag'));
                console.log(myCase && myCase.title);

                // find a suite by title
                const mySuite = helper.find((item, parent) => item.type === 'suite' && item.title.includes('new syntax'));
                console.log(mySuite && mySuite.title);

                // filter failed cases
                const failedCases = helper.filter((item, parent) => item.type === 'case' && item.caseType === 'failed');
                console.log(failedCases.map((it) => it.title));

                // Iterate all items
                helper.forEach((item, parent) => {
                    // do something
                });

            }
        }]
    ]
};

Integration Examples

By using the onEnd hook, we can integrate Playwright report with any other tools, such as:

See playwright-reporter-integrations

Contributing

# Node.js 20+
npm install starfall-cli -g
npm install

npm run build
npm run test

npm run dev

Dependencies