Test run summary time is wildly incorrect

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.

https://cypress.io

MIT License

47.02k stars 3.19k forks source link

Test run summary time is wildly incorrect #29295

Open mitchwd opened 7 months ago

mitchwd commented 7 months ago

Current behavior

The time shown in the text summary at the end of a test is wildly different to the actual time the test took to run.

Running time npx cypress run --component shows two very different times:

 ✔  All specs passed!                        00:39      323      315        -        8        -  

npm run test:component  330.06s user 74.00s system 170% cpu 3:56.71 total

00:39 != 330 seconds

Desired behavior

I expect the summary to show the actual combined test run time.

Test code to reproduce

time npx cypress run --component

Cypress Version

13.7.0

Node version

18.17.1

Operating System

macOS Sonoma 14.4.1 (23E224)

Debug Logs

No response

Other

No response

jennifer-shehane commented 7 months ago

@mitchwd Yah, this is a known issue. The test timing actually only shows the mocha execution times of the tests, so there could have been some other things happening during loading the browser, loading the spec, recording video, uploading to the Cloud (if recording) that aren't accurately captured in this timing.

I'd suggest running in debug mode mode to troubleshoot where timing may be going towards to narrow down the timing difference.

nevercast commented 4 months ago

Fwiw I have a hunch that this is what is causing the truncated video output. My videos are as long as Cypress reports in the duration, which is significantly less than the time it takes for the whole test to run.

I have found that async code, or code that may behave async, can cause this. Especially a request interception.
Using cy.wrap(null).then(() => cy.wait(request).then(intercept => ...)); allowed me to keep my code in sync.
Extra use of cy.wrap(null).then(() => { ... }); can be useful if you're not sure that your function calls are going to have side effects.

After making these changes, the spec that took "4" seconds to run now takes the correct amount of time "15" seconds in Chrome head.

Headless now reports "9" seconds instead of 4, which is still terminating early.

-- Update

Make sure you're respecting the chainables you need to.
Use { scrollBehaviour: false } for some clicks.
Use Chrome instead of Election.

Working fine for me now.