How to instrument and measure unvisited pages?

cherewaty commented 4 years ago

I'm working on a Next.js app where we've just started adding tests with Cypress. Because there's a lot of pre-existing code, I want to use the coverage reports as a progress report to see how we're progressing... but I'm getting a very limited view from the lcov-report.

Pages in the Next.js app that aren't visited in tests do not appear in the report, so I'm getting inflated coverage numbers. Instead of "what percentage of the total statements in the app are covered?", the report gives me "what percentage of the total statements in pages that the tests visited are covered?"

I took a look at https://github.com/kylemh/next-ts-with-cypress-coverage, which led me to adding the all option to the nyc config. Now pages that haven't been touched show up in the lcov-report... but with all zeros.

I've reproduced at https://github.com/cherewaty/next-and-cypress-example/tree/uncovered-page

uncovered.js should instead show 0/1 statements covered

My hypothesis is that Next.js's dev server isn't even attempting to build pages until they're visited, so babel-plugin-istanbul isn't instrumenting them.

Some paths to explore I've been thinking about:

Am I missing some configuration parameter that would make this easy?
Maybe I could hack up the next/babel preset a little bit when NODE_ENV==="test" so all sources get built and instrumented?
Do I need to run cypress against a "production" artifact from next build instead of the dev server?
Could I set up a spot where I import everything just for cypress runs?

Happy to make a PR to this example repo if I get this figured out!

AleksaC commented 3 years ago

I'm facing the same issue. I think you are right about next not building pages until they are visited, because it shows 100% coverage for 404 page even though I don't touch it in the tests. According to docs that page is statically generated at build time, hence the coverage.

bahmutov commented 3 years ago

Yeah not sure - but I guess the tests should visit all pages :)

Sent from my iPhone

On Dec 19, 2020, at 17:02, Aleksa Cukovic notifications@github.com wrote:

I'm facing the same issue. I think you are right about next not building pages until they are visited, because it shows 100% coverage for 404 page even though I don't touch it in the tests. According to docs that page is statically generated at build time, hence the coverage.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

AleksaC commented 3 years ago

Sure, that's the goal. It would just be nice to be reminded that they aren't by failing coverage-check.

bahmutov commented 3 years ago

Can you use the code coverage option “all” with a file pattern to use?

Sent from my iPhone

On Dec 19, 2020, at 17:43, Aleksa Cukovic notifications@github.com wrote:

Sure, that's the goal. It would just be nice to be reminded that they aren't by failing coverage-check.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

AleksaC commented 3 years ago

Here's how I use it:

{
  "all": true,
  "include": [
    "pages/**/*.js",
    "components/**/*.js",
    "constants/**/*.js",
    "utils/**/*.js"
  ],
  "exclude": ["pages/api/__coverage__.js"]
}

Is this what you meant?

bahmutov commented 3 years ago

Yeah and what happens? It is hard to debug anything like this without seeing the code and trying it

Sent from my iPhone

On Dec 19, 2020, at 19:33, Aleksa Cukovic notifications@github.com wrote:

Here's how I use it:

{ "all": true, "include": [ "pages//*.js", "components/*/.js", "constants//*.js", "utils/*/.js" ], "exclude": ["pages/api/coverage.js"] } Is this what you meant?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

AleksaC commented 3 years ago

When I run the tests and get coverage report all works well for the pages that are covered by the tests and for the ones that are not I get a result like the one in the image above in the original issue. Also I have a custm 404 page which is also not touched by the tests but the coverage is reported at 100% for it like I mentioned in my first comment.

I can provide a repo reproducing the issue if you want to take a look.

bahmutov commented 3 years ago

Could you provide a repo - not necessarily your private one but make a fresh public one showing the problem. But could you open an issue in cypress-io/code-coverage repo so it does not get lost? And add a link to this issue for full story

Sent from my iPhone

On Dec 19, 2020, at 19:48, Aleksa Cukovic notifications@github.com wrote:

When I run the tests and get coverage report all works well for the pages that are covered and for the ones that are not I get a result like the one in the image above in the original issue. I can provide a repo reproducing the issue if you want to take a look.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

AleksaC commented 3 years ago

Sure, I was thinking of doing that earlier but I saw this issue so I decided to check here first.

bahmutov / next-and-cypress-example

How to instrument and measure unvisited pages? #115