Add ability to configure ‘upload’ and ‘storage’ of videos separately when test suite passes/fails.

[GrayedFox]

Current behavior:

Currently, it's only possible to switch the video feature on or off in a binary way.

Desired behavior:

To be able to record and process videos only when a test fails. If there's no way to do this (due to the recording feature not knowing whether a test will fail or not when it begins to record) - the processing part could at least be tweaked so that only failed tests have their videos processed and saved, whereas passing tests are technically recorded, but never processed or saved to disk if the test passes.

Versions

Cypress 3.1.0 Chrome 69 Electron 59

[drewbrend]

@GrayedFox This already exists, but the docs are not clear. https://github.com/cypress-io/cypress/issues/2217

[GrayedFox]

And for those who want to disable uploading entirely and still save failed spec videos?

I don't think it makes a lot of sense to bundle this into a single config switch - I can certainly see a use case where a user has a parallel setup and doesn't want to upload any videos, but still wants failed videos to be saved on some machine somewhere. Uploading and recording are two very different things, and I'm now (at the very least) the second person to not understand that the videoUploadOnPasses partly covers this wanted behaviour.

Having two switches makes it super clear:

"uploadVideos": true | false
"recordVideos": true | false | fail | pass

Now, I can save only passing or failing videos, both, or none, and Cypress will upload whatever videos happen to be there.

Edit: I've thrown in "pass" as an option here too, since once a working "only failed tests" switch is working, negating this value based on the config should be trivial, and gives users added functionality

[drewbrend]

I agree that it leaves some functionality to be desired, but the current behaviour seems to do what you and I are asking for.

[GrayedFox]

I might take a look into it then - that is one of the points of open sourcing things I suppose! - if you have any hints for where to start looking, they would be very welcome :+1:

[drewbrend]

Check here: https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/modes/run.coffee#L716

[plemarquand]

@GrayedFox I like your proposal for recordVideos. Currently videos are generated for every suite on my build server but really I only want to keep the ones for failed suites as build artifacts, discarding the videos for passing suites to save space on the build server.

As it stands right now setting videoUploadOnPasses: false is actually worse for my situation, since videos of passing suites are still recorded and saved to disk uncompressed.

[GrayedFox]

Ahoy, sorry I haven't forgotten about this, been pretty busy at work (and a while since I've touched coffee script!) - I think I'll be finished on this other side project soon and dedicate a weekend to this. Of course if someone else wants to jump in feel free 😄

[kjellski]

Heyyy @GrayedFox ,

just wanted to let you know, my ticket was closed by @jennifer-shehane in favor for this. I would really like to get this rolling and wanted to ask you what you think about my PR, #2653 , maybe we can iterate on it to support all the options :D

[GrayedFox]

@brian-mann in response to your comment here - where you ask why this feature is necessary, I think this thread highlights valid use cases, perhaps?

Also I would argue that rather than asking the contributors and users of Cypress why they want to be able to control how they use Cypress, I would ask (aside from having the video recording have default values that you can build other features on top of it): is the video recording feature really so tightly coupled with dashboard functionality that it cannot be disabled without breaking the dashboard?

I'd need to take a better look at the code but that sounds like we may be breaking the single responsibility principle here. Ideally, the dashboard functions as a graphical interface for whatever pluggable features the Cypress user thinks they need... even if you know better and think that real life use cases would necessitate that all people should be using feature X or Y, that doesn't seem to me to be a reason to have such tightly coupled code. That, and using something like feature flags for the dash also means you can rapidly disable any feature which (god forbid!) ships with a serious bug, while working on a hotfix, without disabling other parts of the system that work (just food for thought!).

Thanks again for steering this wonderful product through the murky waters of open source shenanigans - I'm sure comments like these ones bring you immense happiness and joy 🙇

[kjellski]

I'm completely on your page @GrayedFox, very thankful for the cypress project itself and would not be testing my frontends like this if it wouldn't be there...

@brian-mann @jennifer-shehane let's focus on a solution here, the other ticket is closed and probably won't get much attention for others following along currently.

For reference, here are the last two comments on my deprecated issue, cited:

@brian-mann:

@kjellski I'm coming back to this now, and the answer may be in front of me - but I'm confused still as to why this is necessary. As of today, you don't need to record to the dashboard to get value out of the video. The option videoUploadOnPasses only affects when --record. If you're not using the Dashboard then you can manage the video yourself since the videos still be physically written to disk.

I must be missing something. Your comment here: #2652 (comment) suggests that you're not using the Dashboard, and I just want to make sure we're on the same page that you can completely manage the video yourself if that's the case without us needing to change anything on our end.

@kjellski:

Thanks for getting involved again @brian-mann! So the problem in general is this: AFAIK you can not use video recording without sending it out to your servers via upload.

So you would be able to only record videos, without upload? I mean that's what I'm after... but it's not possible to use --record without a key to the dashboard and then it uploads... maybe I'm missing some documentation that suggested this... :D

[dialex]

Is there any update on this feature request?

[marinakuranda]

This feature would be very appreciated. Ideally, it would follow the same logic as with saving screenshots - only saving them on fail.

[seyfer]

Here is another issue, not linked yet to this one, since it was closed. https://github.com/cypress-io/cypress/issues/775

I really want it to happen. We do not need videos for each test on CI, we need video only for failing test, as it works with screenshots.

[Alados]

Any updates about the feature of storing videos (without using Dashboard) only for failed tests?

[kjellski]

@Alados @dialex @marinakuranda this is possible already, but I haven't looked into any updates on the discussions! The funny part is, it was possible from the beginning. You just don't specify the projectId part and set video to true as well as videoUploadOnPasses to false. We use the attached config for us just fine and the videos are just as if you'd run cypress locally. Then just mark them as artifacts in your CI and you have them available without using the cypress dashboard.

I hope this helps :)

{
  "baseUrl": "http://localhost:8080",
  "env": {
    "NODE_ENV": "local"
  },
  "reporter": "junit",
  "reporterOptions": {
    "mochaFile": "test-results/results.[hash].xml"
  },
  "video": true,
  "videoCompression": false,
  "videoUploadOnPasses": false,
  "viewportHeight": 900,
  "viewportWidth": 1440,
  "modifyObstructiveCode": false
}

[dialex]

Well @kjellski, my cypress.json file has that exact configuration (no projectId, "video": true, "videoUploadOnPasses": false) and still Cypress generates videos for all tests, which is not what we want.

To reiterate, we want the ability to record videos only for failed tests. A possible implementation was described at https://github.com/cypress-io/cypress/issues/2522#issuecomment-424358181:

"uploadVideos": true | false
"recordVideos": true | false | fail | pass

I think it's clear by now that the community wants this feature. The specification was already laid out by @GrayedFox. Is there anything blocking the implementation of this feature?

[kjellski]

@dialex aaahhhh, you only want the ones for failed! Sorry - I was not expecting this restriction - just as a workaround, you could just delete the successfull ones with a little script before ending your 'build' for now...

But yes, this seems to be not existing yet AFAIK!

[josefsmrz]

Any updates on this?

[GrayedFox]

@dialex aaahhhh, you only want the ones for failed! ... you could just delete the successfull ones with a little script before ending your 'build' for now...

Ahhhh this old thread! Just to clarify - even if people only want to record videos for failed tests, it's impossible to know if a test will fail or not while the test is still running (since that would mean peering into the future, which not even Google's engineers can do - unless they know something I don't!) - which means recording videos only for failed tests in reality means recording videos for all tests and deleting/handling videos for tests which pass in some way. What @kjellski is suggesting, to me, makes sense.

It would indeed be possible to process a recording and not actually write anything to disk, given enough RAM, which could potentially be more efficient, depending on your/your cloud's setup. Disk space being cheap and SSD write speeds being so blazingly fast nowadays, recording and deleting videos would probably only cost a few milliseconds more per test (theory crafting now, don't quote me on this).

Anywho, TLDR: recording videos only for failed tests means recording videos for all tests, how one handles removing/deleting videos for passing tests is something that is either hidden from the user (Cypress handles it) or not 🤷‍♂️

[josefsmrz]

The point of being able to only save videos of the tests that failed is to avoid processing the videos captured for passing tests as that seems to be the most time-consuming operation in the process.

Don't get me wrong, I too understand that Cypress isn't designed to be a magic orb to see the future with and that I could just delete the videos of passing tests in an after-script, but that way I'm not able to benefit from the speed increase that I would get by not having to process them.

[jennifer-shehane]

Yes, how @GrayedFox explained is how this would need to be implemented.

This issue is still in the 'proposal' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

[mikaelboff]

Waiting for this!!!

[Jmcobb]

People at @shipt would love a feature like this.

[coltdorsey]

Not generating videos for skipped tests would be a welcomed consideration :)

[AskMarvinV]

Waiting for this as well.. :)

[rubiii]

We're also interested in this

[liambutler]

Also interested in this. I suspect that the saving of videos to Jenkins is responsible for our performance issues

[mikaelboff]

any updates on this?

[lucianojss]

I am also interested since at the moment we don't use cypress dashboard, and we keep videos in gitlab-ci artifacts and since our tests are starting to get more and more extensive, the required size to keep all artifacts also grow a lot and main interest is to keep only videos from the failed ones.

[davidmh]

@lucianojss [...] keep only videos from the failed ones.

You can find the videos for failed tests using the screenshot directories since those get generated only for failing cases:

for scenario in $(ls ./cypress/screenshots); do
  cp "cypress/videos/$scenario.mp4" $path_to_your_artifacts
done

[rclmenezes]

@davidmh that's a brilliant hack. Nice

Solution for recursive folders:

for s in $(find $SCREENSHOT_PATH -type f | sed "s|$SCREENSHOT_PATH\/||" | sed "s|\.png||"); do
  mkdir -p $(dirname $TMP_PATH/$s) && cp "$VIDEO_PATH/$s.mp4" $TMP_PATH/$s.mp4
done

[Tomas-Tr]

somebody mastered to implement the hack from @davidmh into CI pipeline which uses copyfiles and publishartifacts task?

[jflayhart]

Hacks and other solutions like "just remove all the non-failed ones" are silly. This is not what we want! We would love a simple config toggle to NOT process SUCCESSFUL videos AT ALL. PERIOD. Regardless of using recording, cypress dashboards, etc.

The problem with said hacks is that it completely doesn't fix the problem nor does it solve optimization issues. A computer's TIME and SPACE is required to process, store, and generate these videos; and this time/space is wasted on videos some of us in the community don't want or need. We use Cypress liberally and the tests take minutes just to process videos we don't need, but there's no way to save on that time unless we disable videos completely, which is also not what we want.

For example, currently 99 tests pass, 1 test fails, but 100 videos are processed. In this example, we only want the ONE FAILED VIDEO RECORDING GENERATED AND SAVED. I think there's still confusion around this ask.

[lizell]

This is mostly a performance issue. Why spend time on encoding videos of passed tests, just to remove them later? All that is needed is a config switch to tell Cypress that we do not care about encoding/keeping the video of a passed test. Much like screenshots already work.

Of course we need to capture the video for each test since we do not know in advance if it will fail or not, but we could skip the time consuming encoding part and saving of the video for a passed test.

[jennifer-shehane]

I've closed the issue about appending (failed) to the video filenames as a duplicate of this issue. #6998 The main motivator of that issue seemed to be what would be achieved in this issue - that people want ~non-passing~ passing tests' videos to be deleted.

If you have a need for the video having a (failed) string appended to the video filename, we'd be open to hearing use cases for this outside of deleting the ~non-passing~ passing videos.

Edited by @jennifer-shehane to clarify people want passing test videos deleted. Typo on my part.

[jflayhart]

The main motivator of that issue seemed to be what would be achieved in this issue - that people want non-passing tests' videos to be deleted.

Thanks for replying @jennifer-shehane I still think there's a misunderstanding here and I'm not sure where the disconnect is (it could be me). It's not that "people want non-passing tests' videos to be deleted", that is in fact the inverse of precisely what we're asking 😄 We don't want failed tests' videos to be removed, we want an opt-in option to have failed tests be the only things processed and saved, in other words, passing tests' videos should not be saved, only failed ones.

[tom10271]

@davidmh that's a brilliant hack. Nice

Solution for recursive folders:

for s in $(find $SCREENSHOT_PATH -type f | sed "s|$SCREENSHOT_PATH\/||" | sed "s|\.png||"); do
  mkdir -p $(dirname $TMP_PATH/$s) && cp "$VIDEO_PATH/$s.mp4" $TMP_PATH/$s.mp4
done

Can you help fixing the bash script so that it supports filename with space? I have tried to add | sed 's/ /\\ /g' but not working. I am not familiar with bash script :(

[tom10271]

@davidmh that's a brilliant hack. Nice Solution for recursive folders:

for s in $(find $SCREENSHOT_PATH -type f | sed "s|$SCREENSHOT_PATH\/||" | sed "s|\.png||"); do
  mkdir -p $(dirname $TMP_PATH/$s) && cp "$VIDEO_PATH/$s.mp4" $TMP_PATH/$s.mp4
done

Can you help fixing the bash script so that it supports filename with space? I have tried to add | sed 's/ /\\ /g' but not working. I am not familiar with bash script :(

I have figured it out. I use TypeScript so it might not works the same as JS user does.

#!/usr/bin/env bash

IFS=

rm -rf ./test-artifacts
mkdir -p ./test-artifacts/videos

chmod -R 777 ./cypress/screenshots
cp -R ./cypress/screenshots ./test-artifacts/screenshots

find ./cypress/screenshots -iname *.ts | sed "s|./cypress/screenshots\/||" | sed "s|\.ts.*||" | while read -r s
do
    number_of_slash=$(echo -n ${s} | tr -cd '/' | expr $(wc -c) | sed 's/^ *//')

    if [ "$number_of_slash" -gt 0 ]; then
        mkdir -p $(dirname ./test-artifacts/videos/$s)
    fi

    cp "cypress/videos/$s.ts.mp4" "./test-artifacts/videos/$s.mp4"
done

[mikaelboff]

We want passed videos not to be processed and saved, any updates on this? @jennifer-shehane

[lizell]

We want failed videos not to be processed and saved, any updates on this? @jennifer-shehane

No, we want videos for passed tests not to be processed (or saved). At least for us a significant amount of time is being spent processing 100s of videos, that we never look at. Only the failed ones are interesting.

[mikaelboff]

We want failed videos not to be processed and saved, any updates on this? @jennifer-shehane

No, we want videos for passed tests not to be processed (or saved). At least for us a significant amount of time is being spent processing 100s of videos, that we never look at. Only the failed ones are interesting.

Oops, sorry, I mistyped, just edited.

[dipasqualew]

I agree that we could save lots of time and CI resources by having the option to just skip processing for videos that we don't care about. Any update on this matter @jennifer-shehane?

Thanks!

[lizell]

Actually, some good news. I tried this maybe some year ago and at the time it didn't work (and the documentation states that it only works for the Dashboard). However I inspected the code and it turns out setting the config key videoUploadOnPasses to false does exactly what we want. It skips compression (most important) and upload (if Dashboard). I just verified it using Cypress 5.3.0.

So @jennifer-shehane, maybe we could just update the documentation (and/or optionally rename the config key)?

[khitrenovich]

Actually, some good news. I tried this maybe some year ago and at the time it didn't work (and the documentation states that it only works for the Dashboard). However I inspected the code and it turns out setting the config key videoUploadOnPasses to false does exactly what we want. It skips compression (most important) and upload (if Dashboard). I just verified it using Cypress 5.3.0.

So @jennifer-shehane, maybe we could just update the documentation (and/or optionally rename the config key)?

I just tried that, and checked the results when one of the four spec files of a run failed. Good news - the videos of the successful specs are not processed (as expected). Bad news - if one of the specs fail, all the videos are uploaded as is, uncompressed, taking x10 more space.

So, using videoUploadOnPasses is not a real solution here.

[lizell]

OK, so for me I have a small script that removes the video files of passed tests before the upload stage. It is kind of a hack though, inspecting the resulting files of the mocha-junit-reporter. In cypress.jsonset something like:

"reporter": "../../node_modules/mocha-junit-reporter",
"reporterOptions": {
  "mochaFile": "../../reports/junit/ui/[hash].xml"
},

and the my hack:

#!/usr/bin/env bash

echo Cleaning up videos:

for REPORT in reports/junit/ui/*.xml
do
  FILE=$(xmllint ${REPORT} --xpath '//@file')
  VIDEO=${FILE/ file=\"integration/tests\/ui\/videos}
  VIDEO=${VIDEO/.js\"/.js}.mp4

  if [ "$(xmllint ${REPORT} --xpath '//@failures>0')" = true ]; then
    echo " Keeping $VIDEO of a test with failure"
  else
    echo "Removing $VIDEO of a passed test"
    rm -f "$VIDEO"
  fi
done

You might want to adjust some paths above and there are certainly a number of other ways this could be achieved, but I leave this here as one way to do it ;).

[dipasqualew]

In all honesty I am starting to wonder the utility of the video feature. Nowadays we all have access to cheap, reproducible pipelines. Personally I run my E2E against a build I can spin up again, and if tests fail I can just point my local Cypress against the site.

However, this is not to say that I think the video feature is working "ok" as of today. While pipelines are cheap, time is not, and being forced to either process all videos, or to upload them, even when they are successful (if any of them fail), is very bad. It adds lots of time in the development-test cycle.

I know the options here are tailored for the Cypress Dashboard, but not everyone uploads these videos there. Major pipelines providers also allow artifact uploads.

[erickwilder]

Actually, some good news. I tried this maybe some year ago and at the time it didn't work (and the documentation states that it only works for the Dashboard). However I inspected the code and it turns out setting the config key videoUploadOnPasses to false does exactly what we want. It skips compression (most important) and upload (if Dashboard). I just verified it using Cypress 5.3.0.

So @jennifer-shehane, maybe we could just update the documentation (and/or optionally rename the config key)?

This doesn't really solve the issue because files are still saved, which means that Cypress spends some resources recording these videos anyway. I would love to have a way (like the one describe by the @GrayedFox ) to choose which videos to record (and save to disk - recordVideos) and whether or not they should be uploaded to the Dashboard (uploadVideos).

Right now I use my own CI-related functions to save artifacts and expose videos internally to the team (not the Dashboard) if a CI pipeline fails. The problem is that even passed test suites includes tons of useless videos in the pipeline artifacts because we can't turn recording off.

[lizell]

But @GrayedFox is requesting exactly that – only failed videos are processed and saved. It is impossible to record only failed tests, since we can not know in advance if a test will fail or not. So what we want is;

• videos are recorded
• failed ones are saved and processed
• successful ones are not processed and discarded

videoUploadOnPasses: false takes us pretty close. But we still need native support to remove passing videos so that we don't have to write "hacks" to do it. We also have our own CI save artifact thingy, so we share the same problem. For us it is solved with the "hack".

[rooby]

In all honesty I am starting to wonder the utility of the video feature. Nowadays we all have access to cheap, reproducible pipelines. Personally I run my E2E against a build I can spin up again, and if tests fail I can just point my local Cypress against the site.

It's definitely useful to save time in debugging tests running in CI pipelines. Running in local cypress is not a viable solution, since sometimes failures occur specifically in the CI pipeline but not locally (for various potential reasons). Also, I don't want to have to run the tests again to investigate when I could instead just have artifacts from the original failed run.

In addition to the issue of time, disk space is unfortunately not unlimited. If your pipelines save their artifacts for a couple of weeks and you have a lot of pipelines running a lot of tests, keeping videos of all passing tests can waste a huge amount of drive space. Especially if you leave those passing test videos unprocessed.

[jennifer-shehane]

Sorry, that was a typo on my part. Everyone jumped on that. 😅 I clarified in my comment that people want passing videos deleted (non-failed videos). https://github.com/cypress-io/cypress/issues/2522#issuecomment-682365040

We do understand what people are asking - that this is an issue with performance, time and resources in CI, regardless of Dashboard use. Sorry for the confusion.

We're evaluating a possible API for more fine grained video configuration.