[Feature] Track Code Coverage via istanbul

mxschmitt commented 3 years ago

Update: It can be easily integrated in the user-land now.

For that follow the guide here.

For any suggestions please comment inside this issue.

barakbs1 commented 3 years ago

We currently use jest-playwrights' code coverage feature and we depend on it. We would love to migrate to new playwright test-runner but we have to have the code coverage feature, and would really hope it would be integrated. Also it would be amazing if there would be an integration with Kibana-Elastisearch for displaying the results. Moreover, we currently use this % test coverage as a bitbucket-pipeline gate, so an integration with bitbucket pipelines would be amazing!

Keep up the amazing work guys! 🚀 🥇

mpuertao commented 3 years ago

Thanks, i need coverage with istanbul. stay tuned

mxschmitt commented 3 years ago

Small update, we created an example guide how you can use it. It is currently there to get an understanding which additional features are necessary to provide a full story around code coverage collections. For any suggestions / feedback it would be highly appreciated to get input especially:

If you are using the babel-plugin-istanbul?
If single browser code coverage like Chromium would be enough for you?
If you would prefer an approach with minified JS + source maps instead of using the babel plugin? (this would require to have the source maps either locally or available on the web-server)
What are you doing with the coverage data? text-summary, uploading it to a provider, serving the html, only for manual checking etc.

mpuertao commented 3 years ago

For me:

If you are using the babel-plugin-istanbul? R:// Yes.
If single browser code coverage like Chromium would be enough for you? R:// Yes, the coverage is a quantitative aspect, and with a single browser I can determine what components I am covering from the E2E tests.
What are you doing with the coverage data? text-summary, uploading it to a provider, serving the html, only for manual checking etc. R:// Generate lcov.info o json o html, we are send this data to elasticsearch and after grafana

anfruiz commented 3 years ago

I implemented the baseFixtures.ts in the test folder and I created my test like this:

import { test } from "./baseFixtures";
test('P', async ({ page }) => {
  await page.goto("http://localhost:3000");
  const commandLine = await page.innerText("p");
  test.expect(commandLine).toBe("Edit src/App.js and save to reload.");
});

mxschmitt commented 3 years ago

How did you add the babel-plugin? This requires a transcompilation step which is not out-of-the box available. This looks like create-react-app and needs by that an ejected installation.

anfruiz commented 3 years ago

This requires a transcompilation step which is not out-of-the box available

I only created the .babelrc file in the root project. That's means that I can't apply coverage if I use creat-react-app or ng new my-app?

mxschmitt commented 3 years ago

create-react-app does not use the babelrc file. For that you would need to either eject your create-react-app or use react-app-rewrired

anfruiz commented 3 years ago

Ok thanks Max, for an angular app is the same case or I can create a babelrc file?

mxschmitt commented 3 years ago

I'm not familiar with Angular, depends how/if they allow you to add a custom babel plugin.

anfruiz commented 3 years ago

Captura de Pantalla 2021-06-28 a la(s) 3 09 03 p m

Hi Max, when I create the baseFixtures.ts in an angular app, it doesn't recognize these 3 dependencies but if I create it in other project the dependencies are ok

mxschmitt commented 3 years ago

hi, this is only a typings warning. It should still be able to work correctly. To get rid of the typing warnings, replacing it with

import * as fs from 'fs';
import * as path from 'path';
import * as crypto from 'crypto';

should fix. Would be awesome if you could verify, then I can adjust it properly.

anfruiz commented 3 years ago

Excellent, this solution fix the warnings!! Thanks!!

dmail commented 3 years ago

According to my tests Istanbul coverage and v8 coverage cannot be merged together. I have opened an issue to track this: https://github.com/istanbuljs/v8-to-istanbul/issues/144. Maybe there is a way but I couldn't make it work.

If confirmed it means coverage collected using Istanbul cannot be merged with coverage returned by page.coverage.stopJSCoverage(). Dropping this here in case you was not aware of this issue.

scottfwalter commented 3 years ago

Excellent, this solution fix the warnings!! Thanks!!

Do you have a sample repo that shows integration with Angular?

mxschmitt commented 3 years ago

Closing since its possible in the user-land, see https://github.com/mxschmitt/playwright-test-coverage

anishkny commented 3 years ago

@mxschmitt It would be great if this were offered as part of Playwright itself via an option instead of in user-land. Thoughts?

anishkny commented 3 years ago

Thanks for reopening. Some points why I think this would be awesome:

Playwright is positioned as primarily a test framework (which can also do other non-test stuff)
A vast majority of modern web apps use babel (React, Vue, etc)
Code coverage is a critical metric for determining test bed health and efficacy
Playwright makes test related stuff super easy (screenshots, videos, browser downloads) and hides the underlying complexities from the user - this feature seems aligned with that

For now, until this feature is enabled, this can of course be achieved in user-land as shown in the demo or by using the playwright-test-coverage package.

scottfwalter commented 3 years ago

I'm struggling to get babel-plugin-istanbul to work with Angular 12 (or any version). Does anyone have an example on how to instrument with Angular? I attempted to set up extra-webpack-config.js like this:

module.exports = {
    module: {
      rules: [
        {
          // test: /\.m?js$/,
        test: /\.ts$/,

          /**
           * Exclude `node_modules` except the ones that need transpiling for IE11 compatibility.
           * Run `$ npx are-you-es5 check . -r` to get a list of those modules.
           */
          exclude: /[\\/]node_modules[\\/](?!(incompatible-module1|incompatible_module_2|some_other_nested_module)[\\/])/,
          use: {
            loader: 'babel-loader',
            options: {
                "plugins": [
                    ["@babel/plugin-proposal-decorators", {"legacy": true}],
                    "babel-plugin-istanbul"
                ],
                "presets": ["@babel/preset-typescript"]
            }
          }
        }
      ]
    }
  };

However, when I run my app I don't have the COVERAGE variable. I think just need to find out how to get babel properly working Angular.

bgotink commented 3 years ago

I'm in a similar situation as @scottfwalter where I'm not using babel or webpack directly but Angular is responsible for the configuration. I'm not planning on hooking/hacking into angular to add babel-plugin-istanbul, as that might break on every angular update.

So I've gone ahead and created a solution that uses v8's coverage instead: https://github.com/bgotink/playwright-coverage. It's very experimental, more of a proof-of-concept at this stage, but it works on a (pretty large) angular application.

To answer the questions of https://github.com/microsoft/playwright/issues/7030#issuecomment-867174653 for our case:

We're not using babel. Using babel would require hacking into webpack configuration we do not control ourselves, which we are not prepared to do.
Only having coverage in chrome is fine for us, as our CI solution currently only runs using chrome anyway due to resource constraints. Even if that were not the case, chromium-only coverage gives more info than no coverage, so I'd still gladly take it
.
Text-summary and HTML report for developers, LCOV file for CI and code analysis

anfruiz commented 3 years ago

@bgotink I'm trying to imlplement your solution but when I execute the test it presents the following error:

(node:79038) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./" in the "exports" field module resolution of the package at .../node_modules/@playwright/test/package.json. Update this package.json to use a subpath pattern like "./*". (Use node --trace-deprecation ... to show where the warning was created) Error: Error while reading @bgotink/playwright-coverage: Cannot find module '.../@bgotink/playwright-coverage'

Can you help me please?

bgotink commented 3 years ago

@anfruiz Could you open a ticket on https://github.com/bgotink/playwright-coverage? I'd rather discuss issues there than continuing in this feature request on the playwright repo, to prevent the discussion here from going off-topic.

stevez commented 2 years ago

Update: It can be easily integrated in the user-land now.

For that follow the guide here.

For any suggestions please comment inside this issue.

I found your repo can't create screen shot, please check the open issue I created in your repo: https://github.com/mxschmitt/playwright-test-coverage/issues/8

cheapCoder commented 2 years ago

@mxschmitt can i use baseFixtures.ts for @playwright/experimental-ct-svelte? I try to extend the context to the test from @playwright/experimental-ct-svelte, and add viteBabel to ctViteConfig in playwright.config.ts, but when using await page.pause() and logging window.__coverage__ in devtool console, i get undefined.

SgtPooki commented 2 years ago

I was able to get the solution mentioned above (https://github.com/microsoft/playwright/issues/7030#issuecomment-867174653) working on a large CRA project after roughly a day of finagling with the various solutions found.

check out the changes at https://github.com/ipfs/ipfs-webui/pull/2014

sh977218 commented 2 years ago

I'm struggling to get babel-plugin-istanbul to work with Angular 12 (or any version). Does anyone have an example on how to instrument with Angular? I attempted to set up extra-webpack-config.js like this:

module.exports = {
    module: {
      rules: [
        {
          // test: /\.m?js$/,
        test: /\.ts$/,

          /**
           * Exclude `node_modules` except the ones that need transpiling for IE11 compatibility.
           * Run `$ npx are-you-es5 check . -r` to get a list of those modules.
           */
          exclude: /[\\/]node_modules[\\/](?!(incompatible-module1|incompatible_module_2|some_other_nested_module)[\\/])/,
          use: {
            loader: 'babel-loader',
            options: {
                "plugins": [
                    ["@babel/plugin-proposal-decorators", {"legacy": true}],
                    "babel-plugin-istanbul"
                ],
                "presets": ["@babel/preset-typescript"]
            }
          }
        }
      ]
    }
  };

However, when I run my app I don't have the COVERAGE variable. I think just need to find out how to get babel properly working Angular.

Hi, do you have a working solution to set up Playwright e2e code coverage for an Angular CLI project? If so, can you please share solution?

CodingProgrammer commented 1 year ago

@mxschmitt Hi mx, this coverage tool can only work within the business project ? Can I separate the e2e automation test from business code?

gauravgandhi1315 commented 1 year ago

Update: It can be easily integrated in the user-land now.

For that follow the guide here.

For any suggestions please comment inside this issue.

@mxschmitt Do you have baseFixture.ts in js? My framework in JavaScript bases

dhythm commented 1 year ago

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul?

It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing.

I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

dhythm commented 1 year ago

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul?

It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing.

I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

Sorry, the above comment was my failure 🤦‍♂️ baseFixture.ts works well for component-testing too.

jeffhuys commented 1 year ago

Sorry, the above comment was my failure 🤦‍♂️ baseFixture.ts works well for component-testing too.

Hey @dhythm, I'm trying to get the same combination working, but it doesn't even work with normal playwright.

I also have vite + vite-plugin-istanbul + playwright(-ct).

In my vite.config.ts, I have:

istanbul({
  include: 'src/*',
  exclude: ['node_modules', 'tests/', 'src/tests/'],
  extension: ['.js', '.ts', '.vue'],
  requireEnv: true,
})

I have the same code as your screenshot, where did you put it? beforeEach and afterEach don't seem to work, and the __coverage__ seems to be undefined (as well as collectIstanbulCoverage()).

While I'm figuring this out, I'm running my tests using npx nyc playwright test tests/integration/login.spec.ts.

Anything I'm missing here?

EDIT: I fixed NORMAL playwright by changing the command I'm running to npx nyc --all --cwd . --include "src/*" npm run test:integration (and then report using npx nyc report --cwd . --reporter=html).

However, when I run the same, but with test:component, I get: page.evaluate: ReferenceError: cov_2is1g4bs0e is not defined. If someone knows why this occurs, please let me know!

    page.evaluate: ReferenceError: cov_2is1g4bs0e is not defined

        at eval (eval at evaluate (:197:30), <anonymous>:1:6)
        at UtilityScript.evaluate (<anonymous>:199:17)
        at UtilityScript.<anonymous> (<anonymous>:1:44)
        at Object.context [as fn] (/Users/jeff/Development/redacted/src/tests/baseFixtures.ts:2:4523)

gselsidi commented 1 year ago

is there an option in playwright config to turn this on?

bradyisom commented 1 year ago

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul? It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing. I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

Sorry, the above comment was my failure 🤦‍♂️ baseFixture.ts works well for component-testing too.

@dhythm, I am in the same position you were. I have my normal Playwright tests working with coverage using vite, vite-plugin-istanbul and baseFixture.ts. However, coverage is not getting collected with the same setup and playwright-ct. Can you please share what your problem was and how you got it working for component testing?

prests commented 1 year ago

@bradyisom do you have an example of how you were able to get normal playwright tests working with vite-plugin-istanbul? I am using vite with vue3 and the coverage report doesn't include any .vue files. Regardless of if you're using vue or not I'd love to see a setup where vite-plugin-stanbul is working with playwright.

I'm not sure why this isn't a feature within playwright to check for instrumented code. Other integration packages like cypress offer this functionality with low effort from the consuming app. Would love to see the same here!

bradyisom commented 1 year ago

@prests, I don't have a simple repro, but there were a few key pieces for me to get it working:

Add the vite-plugin-istanbul plugin to vite.config.js
Add baseFixtures.js See here and use the test and expect exported from there. You can also use the playwright-test-coverage NPM module, as it does pretty much the same thing.
Output for the coverage ends up in the .nyc_output directory after running tests like this. To generate a report, run nyc report after the tests run.

iamdhruv commented 1 year ago

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul? It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing. I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

Sorry, the above comment was my failure 🤦‍♂️ baseFixture.ts works well for component-testing too.

Hey @dhythm, will you be able to share an example repo with your changes, we are struggling to make this work

cenfun commented 1 year ago

Just integrate the code coverage report into monocart-reporter There are 2 supported data inputs:

V8
istanbul

If you are interested, try it according to the examples: 1, https://github.com/cenfun/monocart-reporter-test/tree/main/tests/coverage 2, https://github.com/cenfun/monocart-reporter/tree/main/tests/report-coverage

edumserrano commented 1 year ago

TLDR: see this comment for detailed guidance and code demo on setting up code coverage.

@mxschmitt This issue was opened a couple of years ago and I was wondering if you know what the plans are for this? Meaning, do you plan to provide a better "out of the box" experience to get code coverage?

Thank you for your guide to get code coverage with istanbul. Thanks to it I've been able to get code coverage working in my angular app which is started by playwright's webServer before the tests run.

However, my journey to get code coverage working in Playwright was very painful. I suspect I'm not alone on this. I think a better out of the box experience or at least better docs outlining different ways to get code coverage to work would be a welcome improvement.

I believe @anishkny is right when he says this would be better if it wasn't something only in user-land. 🙏

My Playwright code coverage journey

I started by looking at the official docs on class-coverage and I couldn't get the example in the docs to work for my scenario. My test was just this:

test('basic test', async ({ page }) => {
  if ([test.info](http://test.info/)().project.name === 'chromium') {
    await page.coverage.startJSCoverage();
  }
  await page.goto('/');
  if ([test.info](http://test.info/)().project.name === 'chromium') {
    const jsCoverage = await page.coverage.stopJSCoverage();
    for (const entry of jsCoverage) {
      const converter = v8toIstanbul('', 0, {
        source: entry.source,
      });
      await converter.load();
      converter.applyCoverage(entry.functions);
      const coverageJSON = JSON.stringify(converter.toIstanbul());
    }
  }
});

And my playwright.config.ts had the following:

use: {
  baseURL: "http://127.0.0.1:4999"
},
webServer: {
  command: "npx ng serve --port 4999",
  url: "http://127.0.0.1:4999",
  reuseExistingServer: !_env.CI
}

In terms of the test itself, all is fine, playwright's webServer starts the angular app with ng serve and then is able to navigate to /. The problem is that the line await converter.load(); fails with:

Error: An error occurred while trying to read the map file at c:\dev\repos\temp\my-app\runtime.js.map Error: ENOENT: no such file or directory, open 'c:\dev\repos\temp\my-app\runtime.js.map' at readFromFileMap (c:\dev\repos\temp\my-app\node_modules\convert-source-map\index.js:60:11) at Converter (c:\dev\repos\temp\my-app\node_modules\convert-source-map\index.js:67:32) at Object.exports.fromMapFileComment (c:\dev\repos\temp\my-app\node_modules\convert-source-map\index.js:153:10) at Object.exports.fromMapFileSource (c:\dev\repos\temp\my-app\node_modules\convert-source-map\index.js:165:22) at V8ToIstanbul.load (c:\dev\repos\temp\my-app\node_modules\v8-to-istanbul\lib\v8-to-istanbul.js:52:66) at tests\example.spec.ts:33:23

@cenfun was kind to help me on Playwright's discord channel by answering on my help post Help with code coverage example from the docs. He pointed out that the problem was that v8toIstanbul requires the right source path and linked to [BUG] Code coverage failing with reading map file.

At this point I decided to pivot, I'm not an experienced frontend developer and I didn't feel comfortable trying to do what @cenfun suggested to get the v8toIstanbul working.

I then found this issue and adapted your guide to my Angular project. This again wasn't a straightforward thing for me to do. I had to learn a bit about how Angular doesn't expose it's webpack configuration and possible ways to extend it. I eventually figured it out and was able to extend my Angular app so that it got instrumented via the babel-plugin-istanbul.

At this point I was happy because I got code coverage to work. However, it felt that I had to do too much: 1) install babel-plugin-istanbul 2) update my angular.json to use a different builder so that I could extend Angular's webpack configuration. I decided to use ngx-build-plus for this. 3) create a base fixture with some code that is responsible for recording istanbul's code coverage as per your example. I later found out I didn't need to create this base fixture myself, I could use the Playwright Test Coverage package to do the same. 4) make sure my tests imported the test and expect from the base fixture instead of doing import { expect, test } from "@playwright/test";

The above felt a bit much and it also felt like something I'd have to document well on my project's README because my team/other people in the company would not easily grasp all the moving parts.

Feeling that the above was too much I decided to pivot again and try @cenfun 's monocart-reporter. I eventually found Playwright Component Testing Vue Example and following the instructions on that repo I was able to get monocart-reporter to produce code coverage for my app.

Unfortunately the code coverage from V8 + monocart-reporter wasn't showing accurately. Talking further with @cenfun , he recommended I create an issue for this which can be followed here.

The end of my journey

The whole journey took about 30 hours to complete. I started thinking "this should be simple" and proceeded to hit head-first into a wall of errors and questions of "nothing works".

Now I understand much more about code coverage in general in javascript projects, not just regarding playwright and if I had a lot of this base code coverage knowledge when I started the journey I feel like I could have got things to work much quicker.

However, the point still remains that, at least for developers less experienced with frontend apps, it's way more difficult to get code coverage working than it should be and requires learning/knowing about several concepts. Furthermore, the example in the docs just didn't work at all for my scenario which used the webServer to start my angular app.

microsoft / playwright