search

web-platform-tests / results-collection

Other
41 stars 46 forks source link

Pass browser-specific command line flags when running webrtc/ #125

Open foolip opened 7 years ago

foolip commented 7 years ago

http://wpt.fyi/webrtc is almost entirely red, often because getUserMedia() needs permissions and user interaction. Example: http://wpt.fyi/webrtc/simplecall.html

This is obscuring the “real” status of the tests, and makes it impossible for @guidou, @henbos and other implementers to effectively use the dashboard to prioritize their work.

The "proper" fix for this is:

However, this will at earliest be done some time in early 2018. We should get useful results for the webrtc/ directory ASAP, and then work on paying back that technical debt.

@jgraham FYI. If this will require some changes in wpt runner I hope you think the above approach makes sense.

@patrickkettner @jan-ivar @youennf, we may need your help in picking the right set of command line flags for Edge, Firefox and Safari. Or @agouaillard, you know them, can paste them here?

agouaillard commented 7 years ago

chrome:

copied from https://webrtc.org/testing/ I think we should update that page and keep it current

"""

For manual development and testing, here are some command line flags that are useful for WebRTC-related testing:

--allow-file-access-from-files allows getUserMedia() to be called from file:// URLs.

--disable-gesture-requirement-for-media-playback removes the need to tap a

--use-fake-ui-for-media-stream avoids the need to grant camera/microphone permissions.

--use-fake-device-for-media-stream feeds a test pattern to getUserMedia() instead of live camera input.

--use-file-for-fake-video-capture=path/to/file.y4m feeds a Y4M http://wiki.multimedia.cx/index.php?title=Y4M test file to getUserMedia() instead of live camera input.

"""

missing args we use in KITE:

Note that exceptions can be used to bypass security prompt already ....... https://bugs.chromium.org/p/chromium/issues/detail?id=459532

Firefox:

all through profiles, the main options are:

there are also a few options for H264.

Safari:

most of the settings needs to be done through the command line on the host system, and not as command line arguments to the safari binary. the tokbox team in Sydney got most of them in a shell script: https://github.com/aullman/travis-multirunner/blob/26838e9c6e1765ad31e0b1e118178ac786cca14a/start-safari.sh

from memory all but one are accessible from the shell command line, but one is still only accessible from the dev menus: Screen-Shot-2017-06-16-at-3.20.30-PM.png https://i1.wp.com/webrtcbydralex.com/wp-content/uploads/2017/06/Screen-Shot-2017-06-16-at-3.20.30-PM.png

if you want to use safaritechpreview, there is a flag in selenium: setTechnologyPreview( bool )

@youennf must know, and we can speak about it in spain next week.

youennf commented 7 years ago

Safari version would need to be upgraded first to Safari 11 or STP. The script @agouaillard mentioned should work for Safari11. For STP, it might be good to use some recently added WebDriver options.

agouaillard commented 7 years ago

can you point us to the recently added WebDriver options?

On Wed, Sep 27, 2017 at 10:51 AM, youennf notifications@github.com wrote:

Safari version would need to be upgraded first to Safari 11 or STP. The script @agouaillard https://github.com/agouaillard mentioned should work for Safari11. For STP, it might be good to use some recently added WebDriver options.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/w3c/wptdashboard/issues/125#issuecomment-332567455, or mute the thread https://github.com/notifications/unsubscribe-auth/AAT1nse4h9Z7Oq4kzjO57U4_SsRPwip2ks5smm7-gaJpZM4Plkbi .

-- Alex. Gouaillard, PhD, PhD, MBA

President - CoSMo Software Consulting, Singapore

sg.linkedin.com/agouaillard

-

foolip commented 7 years ago

Thanks @agouaillard, that's very helpful.

@JKereliuk, I think the minimum set of flags for Chrome would be --use-fake-ui-for-media-stream and --use-fake-device-for-media-stream, corresponding to the two WebDriver extensions that are needed.

@youennf, we'd love to have Safari 11 or STP as part of the dashboard. Having them available on Sauce would fix that. I've reached out to y'all over email again.

rwaldron commented 7 years ago

This is excellent—I will do whatever I can to be helpful and offer myself as a resource in achieving these goals.

kereliuk commented 7 years ago

@agouaillard @youennf I'm not sure which options you are talking about? You can pass capabilities to ChromeDriver, or you can pass browser options to the chromeOptions object. Which (if either) do you mean?

jgraham commented 7 years ago

I wonder if it is possible to add the firefox prefs to http://searchfox.org/mozilla-central/source/testing/profiles/prefs_general.js; we already apply those testing prefs when running wptrunner.

Otherwise, if you only want to apply these prefs to the WebRTC tests, the easiest way is to set up the dashboard with some metadata ini files. We already support setting Fx prefs like [1], and I feel like I might have implemented passing binary args at some point for servo, and could certainly make that work for other browsers.

[1] http://searchfox.org/mozilla-central/source/testing/web-platform/meta/webrtc/__dir__.ini#1

agouaillard-cosmo commented 7 years ago

@JKereliuk , @youennf is speaking about safari. I'm speaking about chrome.

We use chromeOptions -> addArgument( ) for each command line flag:

switch (browser.getBrowserName()) {
      case "chrome":
        ChromeOptions chromeOptions = new ChromeOptions();
        chromeOptions.addArguments("use-fake-ui-for-media-stream");
        chromeOptions.addArguments("use-fake-device-for-media-stream");
        capabilities.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
        break;
      case "firefox":
        FirefoxProfile firefoxProfile = new FirefoxProfile();
        firefoxProfile.setPreference("media.navigator.streams.fake", true);
        FirefoxOptions firefoxOptions = new FirefoxOptions();
        firefoxOptions.setProfile(firefoxProfile);
        capabilities.merge(firefoxOptions.toCapabilities());
        break;
      case "MicrosoftEdge":
        capabilities.setCapability("avoidProxy", true);
        break;
      case "safari":
        SafariOptions options = new SafariOptions();
        options.setUseTechnologyPreview(true);
        capabilities.setCapability(SafariOptions.CAPABILITY, options);
        break;
    }
foolip commented 7 years ago

https://github.com/w3c/web-platform-tests/issues/7424 is relevant to this as well.

foolip commented 7 years ago

We should probably include https://wpt.fyi/mediacapture-streams as well, since it would be trivial.

foolip commented 6 years ago

@kereliuk, this is now done for Chrome, but not Firefox, right?

jugglinmike commented 6 years ago

@kereliuk, this is now done for Chrome, but not Firefox, right?

The results collector currently specifies the parameters mentioned above, and, Chrome is reported as passing some WebRTC tests on wpt.fyi. Chrome continues to fail most WebRTC tests, but this matches my experience when manually executing tests hosted on w3c-test.org. Still, I'm not sure if this is an accurate reflection of the implementation status or another issue with the testing environment.

@foolip: to close this issue, I'd like to get confirmation from a Chromium implementer. Could you introduce me to someone who could sign off on the accuracy of the results?

Firefox:

all through profiles, the main options are:

  • media.navigator.streams.fake
  • media.navigator.permission.disabled

I've attempted to set these commands on a "headless" Ubuntu system running Firefox 59.0.2:

xvfb-run --auto-servernum \
  ./wpt run --include webrtc/RTCDTMFSender-ontonechange-long.https.html \
    firefox \
    --setpref 'media.navigator.streams.fake=true' \
    --setpref 'media.navigator.permission.disabled=true'

But that test continues to time out in the environment. This conflicts with the results I receive when running the same browser from my development system and manually visiting https://w3c-test.org/webrtc/RTCDTMFSender-ontonechange-long.https.html, so I suspect there is something else I am missing. @agouaillard do you have any thoughts on this? Can you suggest a Firefox implementer that I could speak with on the topic?