Code coverage broken for Cypress components tests

[cailloumajor]

Software version

OS: Debian GNU/Linux Bullseye Node: 16.18.0 NPM: 8.19.2 / Yarn 1.22.19 Any other software related to your bug: No

What did you get as the error?

No coverage data recorded while running Cypress components tests.

What were you expecting?

Coverage data should have been recorded.

What steps did you take, to get the error?

1. git clone https://github.com/quasarframework/quasar-testing.git && cd quasar-testing
2. git checkout 6ff443f516b71d95e9b577ed82241bab1452a1c3 (v5.0.0-beta.5, no tag for it)
3. cd test-project-vite
4. yarn install
5. yarn sync:cypress
6. yarn test:component:ci

Applying the change below and running again 6. and 7. above, coverage data is recorded well.

diff --git a/packages/e2e-cypress/src/index.js b/packages/e2e-cypress/src/index.js
index 8fb15bb..7fa238c 100644
--- a/packages/e2e-cypress/src/index.js
+++ b/packages/e2e-cypress/src/index.js
@@ -35,7 +35,7 @@ module.exports = async function (api) {
         viteConf.plugins.push(
           istanbul({
             exclude: ['.quasar/*'],
-            forceBuildInstrument: true,
+            forceBuildInstrument: false,
           }),
         );
       });

The setting of forceBuildInstrument was introduced in https://github.com/quasarframework/quasar-testing/commit/d4a5a04041c59b5d5df7894ccd5cf22dfd942373 (PR #281). If I understand, it was to solve problems in e2e tests.

I would be happy to contribute with a PR, but I am not sure of the solution. Should forceBuildInstrument value be dependent of the type of tests running ? In this case, how to know it in the code ?

[IlCallo]

Hey there, I'm currently working on Jest AE v3 beta and will come back in Cypress-land in some week @yusufkandemir do you have any idea or guidace about why this happens?

[yusufkandemir]

forceBuildInstrument makes it so that the plugin is applied only on the build mode of Vite:

https://github.com/iFaxity/vite-plugin-istanbul/blob/a8f86218ecf379a605d6e965250b2b0b396a428a/src/index.ts#L96-L98

  return {
    name: PLUGIN_NAME,
    apply: forceBuildInstrument ? 'build' : 'serve',

However, component testing seems to be using the dev server (serve) at all times. So, the value of forceBuildInstrument should depend on the type of tests as @cailloumajor guessed. One would need to research a bit to see how it can be detected in code.

[IlCallo]

@cailloumajor could you try to change the existing option with forceBuildInstrument: api.ctx.prod ? true : false, and see if this cover all cases? If it does, I'd gladly accept your PR offer :D @yusufkandemir can you check this change still cover your use case?

We add instanbul instrumentation before running Cypress, so we have no way to distinguish between e2e and component testing at that time, we only have info about the build type. There are some obscure npm env variables, but definitely not something I want to rely on

Furthermore, we still want to support instrumentation and coverage for e2e tests running in dev mode, I had no idea forceBuildInstrument acted as a switch

[cailloumajor]

@cailloumajor could you try to change the existing option with forceBuildInstrument: api.ctx.prod ? true : false, and see if this cover all cases? If it does, I'd gladly accept your PR offer :D

@IlCallo: I tested your proposal, it works with both components and e2e tests (running against dev server).

PR #300 created!

@yusufkandemir can you check this change still cover your use case?

I also checked this, with success. I did quite the same as @yusufkandemir in https://github.com/quasarframework/quasar-testing/pull/281#issuecomment-1232871768 (with Quasar CLI installed globally):

$ yarn cross-env NODE_ENV=test start-test "~/.yarn/bin/quasar build && ~/.yarn/bin/quasar serve -H 0.0.0.0 -p 9000 --silent --history dist/spa" http-get://localhost:9000 "cypress run --e2e"

[IlCallo]

Fix released into 5.0.0-beta.6

[radoslavirha]

Hi guys, I can see coverage, but there are zeros in files and my total coverage is wrong. I'm playing with forceBuildInstrument but there is no difference. Can you see this issue too?

https://github.com/quasarframework/quasar-testing/issues/319#issue-1572809922

Thanks