Pass flag '--use-fake...' flags to fake webcam in tests

rbokel commented 5 years ago

Current behavior:

My client code is using the webcam. On ci there is no webcam. For Chrome and Chromium i can fake the webcam using the code below in my plugins/index.js

module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, args) => {
    args.push('--use-fake-device-for-media-stream')
    args.push('--use-fake-ui-for-media-stream')

    return args
  })
}

This makes my tests pass.

On electron my test fails and i cannot seem to pass these options.

On a system without webcam with chrome and the options set as explained above, the test succeeds.

On a system without webcam with electron the test fails.

Desired behavior:

Electron should use the fake-device and fake-ui by default to make tests pass, that require a webcam.

Versions

cypress: 3.1.0 ubuntu 18.04.1 lts chrome 70

Reproducible Example

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta content="stuff, to, help, search, engines, not"
    name="keywords">
  <meta content="What this page is about."
    name="description">
  <meta content="Display Webcam Stream"
    name="title">
  <title>Display Webcam Stream</title>

  <style>
    #container {
      margin: 0px auto;
      width: 500px;
      height: 375px;
      border: 10px #333 solid;
    }

    #videoElement {
      width: 500px;
      height: 375px;
      background-color: #666;
    }
  </style>
</head>

<body>
<div id="container">
  <video autoplay="true"
    id="videoElement">

  </video>
  <div id="statusElement">Status</div>
</div>
<script>
  var video = document.querySelector('#videoElement');
  var statusElement = document.querySelector('#statusElement');

  if (navigator.mediaDevices.getUserMedia) {
    navigator.mediaDevices.getUserMedia({ video: true })
      .then(function(stream) {
        video.srcObject = stream;
        statusElement.innerHTML = '<span>Webcam Success!</span>';
        console.info('Webcam Success!');
      })
      .catch(function(err) {
        statusElement.innerHTML = '<span>Webcam Error!</span>';
        console.error('Webcam Error!');
      });
  }
</script>
</body>
</html>

describe('test-webcam', () => {
  it('should work', () => {
    cy.visit('test-webcam.html');
    cy.get('#statusElement').contains('Webcam Success!');
  });
});

jennifer-shehane commented 5 years ago

~~Hi @rbokel, currently, sending flags through to Electron does not work. There is an issue here to support this: https://github.com/cypress-io/cypress/issues/1519 I will close this issue as duplicate.~~

~~I also created a new issue in our docs to document that passing flags to electron is not currently supported here: {{LINK TO ISSUE}}. Our documentation is open source and contributions are welcome.~~

rbokel commented 5 years ago

Hi @jennifer-shehane, apart from me passing the options, there would also be the possibility for cypress to activate the fake-device by itself, similar to what it does with many other browser features.

Otherwise all tests that involve a webcam will fail. Maybe i should rephrase this ticket ?

jennifer-shehane commented 5 years ago

Yes, I can see this being a valuable feature. I will reopen and rephrase the title. It would be helpful if you could rephrase your original ticket to specify support for these and also any details about how this ends up working - when faking the device.

rbokel commented 5 years ago

Ok, thanks @jennifer-shehane Rephrased it and added a minimal example

jennifer-shehane commented 5 years ago

I've spent some time investigating this issue today on Cypress 3.1.4.

Reproducible repo: https://github.com/jennifer-shehane/webcam-tests

Permission Prompt

Upon attempting to test a webcam, the first hurdle is being prompted by this dialog:

This prompt is discarded by adding the --use-fake-ui-for-media-stream flag.

--use-fake-ui-for-media-stream Bypass the media stream infobar by selecting the default device for media streams (e.g. WebRTC). Works with --use-fake-device-for-media-stream.

Or as WebRTC's Testing guides plainly puts it:

--use-fake-ui-for-media-stream avoids the need to grant camera/microphone permissions.

This should clearly be implemented by default into Cypress.

Live Camera Feed

Upon enabling the --use-fake-ui-for-media-stream and running tests again, the live video feed is presented, which is like "Smile, you're on camera!" 😂

screen-shot-2019-01-04-at-10 05 38-am

This is creepy and likely would not pass in a CI environment without the proper environment set up on the machine to handle a cameras/microphone.

So, we want to be able to have some 'fake' video to test the functionality. There are 2 options here:

--use-fake-device-for-media-stream feeds a test pattern to getUserMedia() instead of live camera input.
--use-file-for-fake-video-capture=path/to/file.y4m feeds a Y4M test file to getUserMedia() instead of live camera input.

When passing --use-fake-device-for-media-stream, a fake video steam is used as shown below in this very low frame rate gif:

When passing --use-file-for-fake-video-capture, there are some requirements outline in Chromium source comments:

Used for testing the video capture pipeline when no real hardware is available. The supported file formats are YUV4MPEG2 (a.k.a. Y4M) and MJPEG/JPEG. YUV4MPEG2 is a minimal container with a series of uncompressed video only frames, see the link http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 for more information on the file format. Several restrictions and notes apply, see the implementation file.

Example Y4M videos can be found in http://media.xiph.org/video/derf. Example MJPEG videos can be found in media/data/test/bear.mjpeg.

Restrictions: Y4M videos should have .y4m file extension and MJPEG videos should have .mjpeg file extension.

Passing the --use-file-for-fake-video-capture flag with the absolute or relative path of the Y4M file looks like this (again in my very low frame rate gif - looks better when running).

Note that in order for the provided video file to work, the previous 2 flags must exist for it to work, so now we have these arguments:

module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, args) => {
    // args.push('--use-fake-device-for-media-stream')
    if (browser.name === 'chrome') {
      args.push('--use-fake-ui-for-media-stream')
      args.push('--use-fake-device-for-media-stream')
      args.push('--use-file-for-fake-video-capture=cypress/fixtures/akiyo_cif.y4m')
    }

    return args
  })
}

Other

The WebRTC testing doc also mentions these flags for testing:

--allow-file-access-from-files allows getUserMedia() to be called from file:// URLs. --disable-gesture-requirement-for-media-playback removes the need to tap a <video> element to start it playing on Android.

I'm going to err on the side of not adding these, because I'm not sure why they would be needed. But please open a new issue if you need one of these flags.

Todo

[x] Add --use-fake-ui-for-media-stream to Cypress default flags
[x] Add --use-fake-device-for-media-stream to Cypress default flags
[x] Write webcam tests
[x] Test that a user pushing arg --use-file-for-fake-video-capture to the browser launch API works properly.

Resources:

Darkensses commented 5 years ago

Hey @jennifer-shehane I cloned your example repo but the y4m video doesn't show :( what version of chrome did you use?

UPDATE: I tested on macOS (High Sierra 10.13.6 / Chrome 71) and the example works very well because the path, so in Windows is a bit different.

Just change the path like this: --use-file-for-fake-video-capture=c:\\path\\to\\video\\akiyo_cif.y4m

jennifer-shehane commented 5 years ago

@Darkensses Thanks for providing a Windows example!

cypress-bot[bot] commented 5 years ago

Released in 3.1.5.

Darkensses commented 5 years ago

wow, that was fast! Thanks 👍

nkrishna79 commented 5 years ago

@jennifer-shehane Hi I have been writing some tests to verify ID verification process during a specific user journey. There are 2 parts to this process - 1. is it to upload a file for example passport and 2. is to capture/upload a selfie image. The problem i am encountering with cypress is that it switches ON the fake camera by default and i am struggling to upload an image or to send out a video file(which I dont want to do) I just want to basically skip this fake camera input and use the upload function. Is there any way I can do this by passing something like "--disable--fake-video-capture". If you need any more info let me know.

jennifer-shehane commented 5 years ago

@nkrishna79 You can upload your own file, instead of the fake file by following these instructions: https://docs.cypress.io/api/plugins/browser-launch-api.html#Use-fake-video-for-webcam-testing

But, basically we pass the --use-fake-ui-for-media-stream and --use-fake-device-for-media-stream flags if you want to play with removing those in the before:browser:launch I linked you to.

nkrishna79 commented 5 years ago

@jennifer-shehane So do you think if I use arguments something along the lines of args.push('--disable-fake-device-for-media-stream') args.push('--disable-fake-ui-for-media-stream') would I still be able to disable the fake camera input? update:

If i pass those arguments to disable it - its not disabling the fake camera input. I tried removing it too but but no luck cypress still forces me to use the fake camera. basically i need a mechanism to disable it. Thats all.

jennifer-shehane commented 5 years ago

No, these args do not exist within Chrome flags. args is an array of flags sent to Chrome containing --use-fake-ui-for-media-stream and --use-fake-device-for-media-stream, so you need to remove those 2 items from the args. So, use JavaScript to find those 2 arguments and remove them from the args array before passing them along.

nkrishna79 commented 5 years ago

Right ok. let me give it a try. Thanks @jennifer-shehane

nkrishna79 commented 5 years ago

No, these args do not exist within Chrome flags. args is an array of flags sent to Chrome containing --use-fake-ui-for-media-stream and --use-fake-device-for-media-stream, so you need to remove those 2 items from the args. So, use JavaScript to find those 2 arguments and remove them from the args array before passing them along.

@jennifer-shehane I tried some methods to disable it. But couldnt get around on how to do this via JavaScript. Would you be able to show me a sample step on how to do this via JS. Thanks

nkrishna79 commented 5 years ago

@jennifer-shehane I can see that its been passed as flag to chrome from this script https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/browsers/chrome.coffee#L21

But how/where can I comment out this line in order for me to be able to not use the fake camera ui?

vacekj commented 4 years ago

Solution for providing mjpeg file on Windows (MJPEG has much better compression than y4m)

Convert an mp4 file using ffmpeg: ffmpeg -i input.mp4 output.mjpeg
Move the file to cypress/plugins
in cypress/plugins/index.js add the following snippet to module.exports

on("before:browser:launch", (browser = {}, args) => {
        if (browser.name === "chrome") {
            args.push("--use-fake-ui-for-media-stream");
            args.push("--use-fake-device-for-media-stream");
            args.push(`--use-file-for-fake-video-capture=${__dirname}\\out.mjpeg`);
        }

        return args;
    });

Using __dirname makes sure we get the absolute path Windows needs

Profit!

Original question: Hi, I have made with work with a .y4m file, however, when using a .mjpeg file encoded from an .mp4, it fails. Are there any other requirements for the .mjpeg format apart from the one stated? (must end with .mjpeg, which it already does) Thanks.

Berkmann18 commented 4 years ago

@JouzaLoL I tried your solution (adapted to GNU/Linux) and Cypress doesn't show the video on my test suite.

@jennifer-shehane

Chromium source comments:

Example MJPEG videos can be found in media/data/test/bear.mjpeg.

The media/data directory is gone from there and it seems that the example videos were also removed.

I've tried your example (on cypress@4.3.0) after following https://docs.cypress.io/api/plugins/browser-launch-api.html#Use-fake-video-for-webcam-testing and none of the videos I tried, were shown on the webcam view of the app (with both --use-fake-... and the --use-file-for-fake-video-capture flags being pushed to launchOptions.args), what could be causing that?

soduor commented 4 years ago

Is it possible to see actual fake video play-back in electron?

lzgrzebski commented 4 years ago

@jennifer-shehane similar question like above but Im trying to use fake video but in firefox. in selenium there will be smth like options.addPreference("media.navigator.streams.fake", true);

Darkensses commented 4 years ago

@jennifer-shehane similar question like above but Im trying to use fake video but in firefox. in selenium there will be smth like options.addPreference("media.navigator.streams.fake", true);

However, firefox doesn't support an injection video like y4m or another format: https://github.com/mozilla/geckodriver/issues/1429

thisismydesign commented 3 years ago

I'm also having trouble making a custom video show up. I'm on WSL2 using the Linux path format: launchOptions.args.push('--use-file-for-fake-video-capture=cypress/fixtures/folder/file.mjpeg'). I'm getting the default green video.

thisismydesign commented 3 years ago

In fact I can mistype the file name and get the same result. It would be really helpful to get an explicit error if the referenced file is not found/recognized. Hard to figure out what the issue might be otherwise.

Darkensses commented 3 years ago

I'm also having trouble making a custom video show up. I'm on WSL2 using the Linux path format: launchOptions.args.push('--use-file-for-fake-video-capture=cypress/fixtures/folder/file.mjpeg'). I'm getting the default green video.

hey @thisismydesign I recommend you using a y4m video. When the video file is not found/recognized, chrome display the 'green video' by default. Please try with a y4m file :)

thisismydesign commented 3 years ago

@Darkensses Thanks, in the end, y4m didn't work for me, and actually mjpeg did. As described in https://github.com/cypress-io/cypress/issues/9562

vikrant-lh commented 3 years ago

Is there a way to remove this arg while launching chrome? https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/browsers/chrome.ts#L100

jennifer-shehane commented 3 years ago

@vikrant-lh You can modify the browser launch args. See https://on.cypress.io/browser-launch-api#Modify-browser-launch-arguments-preferences-and-extensions

vikrant-lh commented 3 years ago

@jennifer-shehane Thanks, I got a workaround, I've edited the array to remove un-required flags :) thanks :)

cypress-io / cypress