search

cypress-io / code-coverage

Saves the code coverage collected during Cypress tests
MIT License
430 stars 108 forks source link

cy.task('combineCoverage') timed out after waiting 60000ms getting this error during bitbucket CI build. #455

Open Ppkd2021 opened 3 years ago

Ppkd2021 commented 3 years ago

Logs and screenshots cy.task('combineCoverage') timed out after waiting 60000ms. getting this error a lot during the execution of test cases on bitbucket but locally when the same test cases are executed then it is working fine. The above error is randomly thrown for various test cases.

Versions Version - 3.9.2 Cypress version - 7.3.0

Describe the bug A clear and concise description of what the bug is.

Link to the repo Bugs with a reproducible example, like an open source repo showing the bug, are the most likely to be resolved.

Example image

Ppkd2021 commented 3 years ago

@bahmutov Please can you help us in resolving this issue as due to this it is costing us a lot of build time in bitbucket, If one test case fails due to the above error then we need to re-run all the test cases in bitbucket that is another challenge

bahmutov commented 3 years ago

Can you create a reproducible example for me to debug this?

Sent from my iPhone

On Jun 16, 2021, at 07:59, Ppkd2021 @.***> wrote:

 @bahmutov Please can you help in resolving this issue as due to this it is costing us a lot of build time in bitbucket. If one test case fails due to the above error then we need to re-run all the test cases in bitbucket that is another challenge.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Ppkd2021 commented 3 years ago

@bahmutov The issue is not happening with one or two specific test cases, It is random in nature.
.

dvkruchinin commented 3 years ago

Hi @bahmutov, I sometimes encounter such an error in GitHub Actions CI. Can you please suggest the debug settings when running Cypress tests to get the correct output in case of an error?

bahmutov commented 3 years ago

Enable the debut logs via an environment variable DEBUG?

Sent from my iPhone

On Jul 20, 2021, at 10:45, Dmitry Kruchinin @.***> wrote:

 Hi @bahmutov, I sometimes encounter such an error in GitHub Actions CI. Can you please suggest the debug settings when running Cypress tests to get the correct output in case of an error?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

dvkruchinin commented 3 years ago

Enable the debut logs via an environment variable DEBUG? … Sent from my iPhone On Jul 20, 2021, at 10:45, Dmitry Kruchinin @.***> wrote:  Hi @bahmutov, I sometimes encounter such an error in GitHub Actions CI. Can you please suggest the debug settings when running Cypress tests to get the correct output in case of an error? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Yes. Probably the inclusion of a full debug will not give the necessary result? Or need to use DEBUG=cypress:*?

bahmutov commented 3 years ago

Look at the debug section in the readme

Sent from my iPhone

On Jul 20, 2021, at 10:58, Dmitry Kruchinin @.***> wrote:

 Enable the debut logs via an environment variable DEBUG? … Sent from my iPhone On Jul 20, 2021, at 10:45, Dmitry Kruchinin @.***> wrote:  Hi @bahmutov, I sometimes encounter such an error in GitHub Actions CI. Can you please suggest the debug settings when running Cypress tests to get the correct output in case of an error? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Yes. Probably the inclusion of a full debug will not give the necessary result? Or need to use DEBUG=cypress:*?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

dvkruchinin commented 3 years ago

Look at the debug section in the readme … Sent from my iPhone On Jul 20, 2021, at 10:58, Dmitry Kruchinin @.> wrote:  Enable the debut logs via an environment variable DEBUG? … Sent from my iPhone On Jul 20, 2021, at 10:45, Dmitry Kruchinin @.> wrote:  Hi @bahmutov, I sometimes encounter such an error in GitHub Actions CI. Can you please suggest the debug settings when running Cypress tests to get the correct output in case of an error? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Yes. Probably the inclusion of a full debug will not give the necessary result? Or need to use DEBUG=cypress:*? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

The error was reproduced. But the reason is still not clear. https://github.com/dvkruchinin/cvat/pull/156/checks?check_run_id=3120924930

chasemgray commented 2 years ago

@bahmutov We stopped using cypress code coverage due to this issue. What we found is that on longer tests, the method which stringifies and parses the code coverage stats can get slower and slower and eventually time out.

bahmutov commented 2 years ago

:(

Sent from my iPhone

On Jan 6, 2022, at 19:56, Chase Gray @.***> wrote:

 @bahmutov We stopped using cypress code coverage due to this issue. What we found is that on longer tests, the method which stores and retrieves the code coverage stats can get slower and slower and eventually time out.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

Fuhji commented 2 years ago

We are having the same issue, any workarounds re: strinfigy are out of date. Unless theres some workaround or fix we'll also have to stop using cypress code coverage

raysunqi commented 2 years ago

+1 having the same issue on gitlab ci builds with 4gb ram, local works fine, increasing 'taskTimeout' to 5 mins doesn't help

dane-harnett commented 2 years ago

We have also been experiencing this issue and my investigation found the following:

Commenting out (and adjusting the variable names) the following lines seemed to resolve the issue: https://github.com/cypress-io/code-coverage/blob/master/support.js#L17-L18

These lines filter out the coverage results from the spec/test files.

For some reason these lines get slower and slower each time they process the coverage for each test in our suite.

For reference our resulting coverage output when stringified (excluding the spec/test files) is about 1MB.

Some potential solutions that I can think of:

  1. Add a config option to skip stripping out the spec/test file results
  2. Improve the performance of the filter functions
dane-harnett commented 2 years ago

For some more info:

Our coverage object contains ~4500 keys. Each check by minimatch takes about 3.5ms. So in total it almost takes 20s for this part of the process.

I'm unsure why it eventually gets to >60s but after about three tests it will eventually fail.

eulersson commented 2 years ago

I'm facing the same issue as well. How can I provide more information like the number of keys, when running from CI?

I think I have 244 keys and this problem happens on my AWS EC2 T3 Medium (4 GB RAM).

If I cy.pause() when running locally and then check the coverage object I get 244, I don't know if this is relevant, compared to @dane-harnett 's 4500 it's nothing.

Object.keys(window.__coverage__).length
244

This error only happens on CI, never seen it when running it locally.

eulersson commented 2 years ago

Here's a bit of an update that seemed to work for me, but sadly on the resources side, not the plugin.

I launched 3 pipelines in GitLab and they all passed with coverage enabled after changing the amount of CPU my docker runner uses.

My runner (T3 Medium on AWS which as 2 vCPU and 4GB RAM) was limited to use 1 CPU in the GitLab Runner configuration because I wanted to have room for SSHing and debugging, but now I increased it to 2, that is, the full compute power of the machine, and all pipelines passed.

So be aware that the issue might be related to CPU and not so much about RAM.

Those are my speculations though...

eulersson commented 2 years ago

False positive. I'm still getting those errors again. I'm using Nx for my project management.

artipsingh commented 1 year ago

Is there any update on this issue? we've started to experience this error during deployments on our CI runs

aantes-st commented 2 months ago

For me this task fails when the coverage object is too large. The task never makes it across to the node process, leading me to believe there is a limit to the size of the string you can pass through cy.task. I wonder if this could be updated to send coverage in a series of batches?