search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
47.1k stars 3.19k forks source link

Record videos after first test failure - to capture video only during retries #16681

Open xtroncode opened 3 years ago

xtroncode commented 3 years ago

What would you like?

Currently when we enable video recording, videos are recorded for all the tests (though we can choose to upload only the failed ones). Due to this the disk IOPS usage when running the tests is very high. It would be great if we have an option to record videos only when a test is being retried due to failure.

Why is this needed?

This should help reduce disk iops usage significantly.

jennifer-shehane commented 3 years ago

You can do this already with this workaround by deleting the videos you don't want to keep: https://docs.cypress.io/api/plugins/after-spec-api#Delete-the-recorded-video-if-no-tests-retried

Duplicate of https://github.com/cypress-io/cypress/issues/16377

xtroncode commented 3 years ago

@jennifer-shehane thank's for the reply but I think I haven't been able to explain my request very well. Let me try again.

Currently when we enable video recording, videos are recorded for all the tests (though we can choose to upload only the failed ones). Due to this the disk IOPS usage when running the tests is very high.

What I am requesting in that tests should be recorded only and only when a test has failed and is being retried. I don't want to be able to delete them, I want them to not be recorded at all.

Let me know if this still does not clarify my request.

Thanks.

jennifer-shehane commented 3 years ago

Hey @xtroncode, the video recording cannot record something that has already happened in the past if the video recording is completely turned off initially. If the tests are not being recorded for video, then a test fails - then it's too late to turn on the video recording because the test has already failed.

There is increased CPU used to record video during the entire test run and we're looking into ways to improve this performance overall. If you'd facing performance issues due to the video recording and running Cypress and whatever else is running in the background, we suggest looking at increasing the resources on the machine.

xtroncode commented 3 years ago

@jennifer-shehane I only want to record the retry and not the first failure..

jennifer-shehane commented 3 years ago

Hmm, we haven't discussed the implementation of turning on video recording mid-run before. So I'm not totally sure how we could achieve this. But an interesting concept.

AlexDaniel commented 3 years ago

Found this issue while trying to figure out the same. We have lots of tests, I don't want to load the runners (both disk and CPU) by video recording when it is not needed (it just makes the runs slower), but I do want to see how something failed if it did. There should be an option to only record video of retries.

mfittko commented 2 years ago

I second this, tests are also becoming much slower and with the retry option enabled on CI, we're only interested in videos of failing retries

ghost commented 2 years ago

This would be a nice thing to have. Right now there is a huge performance penalty we have to pay for having videos recorded all the time by default expecting that a test might fail. Would be nice if the video recording can only happen for retries. Granted that maybe a retried test can pass while the original run failed, but this is something people can live with considering not recording for the 1st run will give a good performance boost overall.

paleite commented 2 years ago

Doubled performance boost...?!

@xtroncode This is a great idea!

I just did the math, and I would really like to see something like this. When we've used Cypress at my company with vs. without recording videos we've gone from 21m10s to 10m0s.

What I like most about this idea is that it would improve the Cypress-performance by 50% at our company: Normally, we would get best-case times of 10m0s and usually it's 1 or 2 tests that need to be retried in case of failures, which would take about 20-30s longer. So if this feature was implemented, failed runs would usually take about 10m30s AND we would receive videos for the retried runs, basically doubling Cypress' speed.

@jennifer-shehane I think it's great that you've added this to your backlog, and personally, I think this sounds like one of the biggest performance boosts you'll be able to give your customers. If you can, please put it higher in priority. I mean, which team DOESN'T want to double the speed of their e2e-pipeline by changing a single line of config, right?

sszemer commented 2 years ago

just stumbled on this while trying to improve cpu loads. pls make this happen as it is (like @paleite said) a solid 50% boost

brunoargolo commented 2 years ago

In our case a hello world cypress test with video off takes 12 seconds to initialize and run, all the test does is console.log('hello'); Turning video on brings it to 60 seconds and produces a 45 second video clip with a mostly blank page.

This feature would be a great way to have major performance gains.

@jennifer-shehane , as an extra, if I understand correctly the video is created by running ffmpg on a bunch of screenshots. you could still "record" the screenshots but only trigger ffmpg if the test fails at the end. Although for thouse concerned with IOPS, it might actually increase it, so it could be an optional feature...

AlexDaniel commented 2 years ago

@brunoargolo it also sounds to me like you're suffering from https://github.com/cypress-io/cypress/issues/18564, https://github.com/cypress-io/cypress/issues/22868 or https://github.com/cypress-io/cypress/issues/23830, or all those combined.

cp-michael-schoormann commented 1 year ago

I would love this feature as well, pls reconsider the priority in the backlog. We have also perfomance issue with videorecording on when running the tests in ci pipeline.

brunoargolo commented 1 year ago

@cp-michael-schoormann I ended up migrating to Playwright. It does have a feature to record only on retries and maybe due to this it's at least twice as fast than cypress. You loose the little left menu that cypress has but it was worth it for us.

elevinixl commented 1 year ago

Definitely want to +1 this request for performance reasons.

t00 commented 1 year ago

This feature would be brilliant. It would save time, cloud running cost, cypress video storage costs etc. If you could squeeze in a similar feature of disabling log on first run (but enabling on retries), cypress performance would be likely on par with playwright for successful tests.

rafawalter commented 12 months ago

Similar to https://github.com/cypress-io/cypress/issues/27653