Coverage Overwritten after each testsuite

Bmahecha123 commented 5 years ago

Hello!

I have a StencilJS project (version 0.14.0) and they have started using Puppeteer, they have provided a small library to implement Puppeteer tests. I have implemented your dependency the way you have documented it. However if I do the same setup in each test file the previous coverage appears to get overwritten. This is reflected when I generate the nyc report. Here is my general setup for each test file.

`import { newE2EPage, E2EPage } from '@stencil/core/testing'; import * as pti from 'puppeteer-to-istanbul';

describe('my-component', () => { let page: E2EPage;

beforeEach(async () => { page = await newE2EPage(); await page.coverage.startJSCoverage(); });

afterEach(async () => { const jsCoverage = await page.coverage.stopJSCoverage();

pti.write(jsCoverage);

});`

Please let me know if there is any more information that you need. Thank you for your time, speak to you soon hopefully!

ps011 commented 5 years ago

Maybe you can use beforeAll and afterAll

joe223 commented 4 years ago

Every Page instance would generate a new puppeteerFormat. How could we merge multiple puppeteerFormat data?

Ref: https://github.com/istanbuljs/puppeteer-to-istanbul/blob/c57bd741534813a7b42c8f300ddb61ef42086ef1/index.js#L4

Update We should set jsonPart as a top scoped variable in puppeteer-to-istanbul, so every page instance test coverage record could be stored. https://github.com/istanbuljs/puppeteer-to-istanbul/blob/de9109c794523eb8924ef24770503908650cc952/lib/puppeteer-to-istanbul.js#L35-L45

Besides iterator should be same as jsonPart https://github.com/istanbuljs/puppeteer-to-istanbul/blob/de9109c794523eb8924ef24770503908650cc952/lib/output-files.js#L62-L66 And, should not save file as ${parsedPath}-${this.iterator}.js if it is inline.

Update This is the fix PR https://github.com/istanbuljs/puppeteer-to-istanbul/pull/42

sebnitu commented 4 years ago

Was this ever resolved or did anyone find a workaround? I'm having the same issue where I can't find a way to keep all my results from multiple test suites. Do we just have to write all E2E tests in a single suite to get coverage with puppeteer-to-istanbul?

joe223 commented 4 years ago

@sebnitu Hi, what's the version you had used? And, would you mind give us a repo for producing this issue?

BTW, make sure that you didn't forget @next tag with installation.

npm i puppeteer-to-istanbul@next

sebnitu commented 4 years ago

@joe223 Thanks for the reply!

You can see this issue in my branch here: https://github.com/sebnitu/scroll-stash/tree/add-coverage-to-puppeteer

This script should reproduce the issue: npm run test:integration

When I install using @next I get v1.3.0, is that correct? Regardless, the issue is the same using v1.4.0.

I currently have 4 integration test suites and depending on which one finishes last, I get that suites coverage results. This is consistent when I run each file individually and the result doesn't change when they're ran together.

Current implementation

Each suite has the following setup:

beforeAll(async () => {
  await page.coverage.startJSCoverage();
  // other stuff..
});

afterAll(async () => {
  const jsCoverage = await page.coverage.stopJSCoverage();
  pti.write(jsCoverage);
});

joe223 commented 3 years ago

I guess you should create an independent Page instance to collect coverage data. @sebnitu

eg:

describe('Wheel Action', () => {
    let page = null

    before(async function () {
        page = await browser.newPage()
        await page.coverage.startJSCoverage()
        await page.goto(global.entryPath)
    })

   after(async function () {
        const [jsCoverage] = await Promise.all([
            page.coverage.stopJSCoverage()
        ])

        pti.write([...jsCoverage])
    })
    // balabala

babaosoftware commented 3 years ago

@joe223 @sebnitu The "problem" is in https://github.com/istanbuljs/puppeteer-to-istanbul/blob/master/lib/puppeteer-to-istanbul.js, line 29: fs.writeFileSync(outFilePath, '') where the out.json file is truncated every time pti.write is called. So depending where you call it, or how many test/test suites you have, you will get only the results for the last one that called pti.write.

babaosoftware commented 3 years ago

@sebnitu To merge coverage information from different test suites, you can do this:

In your afterAll, when calling pti.write, pass a specific name for the json file like this: await pti.write([...jsCoverage, ...cssCoverage], {outputName: 'my-out-file' }); This will create a file my-out-file.json in .nyc-output directory, just for this test suite.
After all test suites have run, execute nyc merge .nyc_output ./.nyc_output/coverage.json to merge all the files in one (.nyc_output is the directory where coverage results are written)
Run nyc report to generate the coverage report.

babaosoftware commented 3 years ago

My suggestion with passing outputName was probably based on a forked version I was working on. Basically I created my own option for PuppeteerToIstanbul, to override the out.json file name, much like it's done for the storagePath, "./.ny_output". Maybe I'll open a pull request for it.

pea-nut-z commented 3 years ago

Every time you open a new page, the coverage gets overwritten. If you keep all the testings on the same page then this wouldn't be a problem. For example, open new page and start coverage in your first describe block's beforeAll function. Then, close page and end coverage in your last describe block's afterAll function. If you need to reset your page in each describe, you could do a refresh page instead of opening a new page that erases previous coverage. That's a simple solution for people who don't care to start new page in every test block. I did that for my project below. https://github.com/pea-nut-z/sneakers-image-classification

istanbuljs / puppeteer-to-istanbul

Coverage Overwritten after each testsuite #20

Current implementation