Best way to access unsupported WD API's from web_sys?

emiddleton commented 4 years ago

Summary

What is the best way to access an API that is not currently supported by web_sys. I am tying to access the Screen Capture API[1] which is still a working draft.

Additional Details

I am trying to use the screen capture API[1] which extends the MediaDevices[2] with the getDisplayMedia() function that returns a Promise similar to the way getUserMedia() works. As this is a working draft API I understand it should not be added to web_sys yet. What is the best way to access this API.

Pauan commented 4 years ago

Somebody (possibly you) has to add the WebIDL to the unstable folder, by following these directions.

emiddleton commented 4 years ago

Thanks @Pauan I will take a look at adding the WebIDL. In the mean time I found the following workaround.

#[wasm_bindgen]
extern "C" {
    # [ wasm_bindgen ( extends = :: js_sys :: Object , js_name = DisplayMediaStreamConstraints ) ]
    #[derive(Debug, Clone, PartialEq, Eq)]
    pub type DisplayMediaStreamConstraints;

    #[wasm_bindgen (extends = MediaDevices, extends = :: js_sys ::Object, js_name = MediaDevices, typescript_type = "MediaDevices")]
    #[derive(Debug, Clone, PartialEq, Eq)]
    pub type MediaDevices2;

    #[wasm_bindgen ( catch , method , structural , js_class = "MediaDevices" , js_name = getDisplayMedia ) ]
    pub fn get_display_media(this: &MediaDevices2) -> Result<::js_sys::Promise, JsValue>;

    #[wasm_bindgen ( catch , method , structural , js_class = "MediaDevices" , js_name = getDisplayMedia ) ]
    pub fn get_display_media_with_constraints(
        this: &MediaDevices2,
        constraints: &DisplayMediaStreamConstraints,
    ) -> Result<::js_sys::Promise, JsValue>;
}

arn-the-long-beard commented 3 years ago

Hello guys :)

When looking on documentation at https://developer.mozilla.org/en-US/docs/Web/API/Screen_Capture_API and at https://rustwasm.github.io/wasm-bindgen/api/web_sys/struct.MediaDevices.html it seems we still do not have getDisplayMedia() available in our web_sys .

I was wondering if there was a plan to add it so we can make screenshot easily in Rust :smile:

I do not understand the instructions there https://github.com/rustwasm/wasm-bindgen/pull/2000#issuecomment-593699351 . It is a bit far from my skills right now.

If there is no plan for it, how could I use it ?

MartinKavik commented 3 years ago

I do not understand the instructions there #2000 (comment) . It is a bit far from my skills right now. @arn-the-long-beard

I would add https://www.w3.org/TR/screen-capture/#mediadevices-additions to https://github.com/rustwasm/wasm-bindgen/blob/master/crates/web-sys/webidls/enabled/MediaDevices.webidl so it looks like https://searchfox.org/mozilla-central/source/dom/webidl/MediaDevices.webidl.
Then I would run commands from https://github.com/rustwasm/wasm-bindgen/pull/2000#issuecomment-593699351.
Then I would try to link the local web_sys to my project and try if the new API works as expected.
Then I would read CONTRIBUTING.md, update CHANGELOG.md if needed and create a PR to this repo with the link to this issue.

arn-the-long-beard commented 3 years ago

Well, I tried to make the change :

/* This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this file,
 * You can obtain one at http://mozilla.org/MPL/2.0/.
 *
 * The origin of this IDL file is
 * http://dev.w3.org/2011/webrtc/editor/getusermedia.html
 *
 * Copyright © 2012 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C
 * liability, trademark and document use rules apply.
 */

[Func="Navigator::HasUserMediaSupport"]
interface MediaDevices : EventTarget {
  [Pref="media.ondevicechange.enabled"]
  attribute EventHandler ondevicechange;
  MediaTrackSupportedConstraints getSupportedConstraints();

  [Throws, NeedsCallerType]
  Promise<sequence<MediaDeviceInfo>> enumerateDevices();

  [Throws, NeedsCallerType]
  Promise<MediaStream> getUserMedia(optional MediaStreamConstraints constraints);

  // We need [SecureContext] in case media.devices.insecure.enabled = true
  // because we don't want that legacy pref to expose this newer method.
  [SecureContext, Pref="media.getdisplaymedia.enabled", Throws, NeedsCallerType, UseCounter]
  Promise<MediaStream> getDisplayMedia(optional DisplayMediaStreamConstraints constraints = {});
};

I did the commands and got :

#![allow(unused_imports)]
use super::*;
use wasm_bindgen::prelude::*;
#[wasm_bindgen]
extern "C" {
    # [wasm_bindgen (extends = EventTarget , extends = :: js_sys :: Object , js_name = MediaDevices , typescript_type = "MediaDevices")]
    #[derive(Debug, Clone, PartialEq, Eq)]
    #[doc = "The `MediaDevices` class."]
    #[doc = ""]
    #[doc = "[MDN Documentation](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices)"]
    #[doc = ""]
    #[doc = "*This API requires the following crate features to be activated: `MediaDevices`*"]
    pub type MediaDevices;
    # [wasm_bindgen (structural , method , getter , js_class = "MediaDevices" , js_name = ondevicechange)]
    #[doc = "Getter for the `ondevicechange` field of this object."]
    #[doc = ""]
    #[doc = "[MDN Documentation](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/ondevicechange)"]
    #[doc = ""]
    #[doc = "*This API requires the following crate features to be activated: `MediaDevices`*"]
    pub fn ondevicechange(this: &MediaDevices) -> Option<::js_sys::Function>;
    # [wasm_bindgen (structural , method , setter , js_class = "MediaDevices" , js_name = ondevicechange)]
    #[doc = "Setter for the `ondevicechange` field of this object."]
    #[doc = ""]
    #[doc = "[MDN Documentation](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/ondevicechange)"]
    #[doc = ""]
    #[doc = "*This API requires the following crate features to be activated: `MediaDevices`*"]
    pub fn set_ondevicechange(this: &MediaDevices, value: Option<&::js_sys::Function>);
    # [wasm_bindgen (catch , method , structural , js_class = "MediaDevices" , js_name = enumerateDevices)]
    #[doc = "The `enumerateDevices()` method."]
    #[doc = ""]
    #[doc = "[MDN Documentation](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/enumerateDevices)"]
    #[doc = ""]
    #[doc = "*This API requires the following crate features to be activated: `MediaDevices`*"]
    pub fn enumerate_devices(this: &MediaDevices) -> Result<::js_sys::Promise, JsValue>;
    # [wasm_bindgen (catch , method , structural , js_class = "MediaDevices" , js_name = getDisplayMedia)]
    #[doc = "The `getDisplayMedia()` method."]
    #[doc = ""]
    #[doc = "[MDN Documentation](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia)"]
    #[doc = ""]
    #[doc = "*This API requires the following crate features to be activated: `MediaDevices`*"]
    pub fn get_display_media(this: &MediaDevices) -> Result<::js_sys::Promise, JsValue>;
    #[cfg(feature = "MediaTrackSupportedConstraints")]
    # [wasm_bindgen (method , structural , js_class = "MediaDevices" , js_name = getSupportedConstraints)]
    #[doc = "The `getSupportedConstraints()` method."]
    #[doc = ""]
    #[doc = "[MDN Documentation](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getSupportedConstraints)"]
    #[doc = ""]
    #[doc = "*This API requires the following crate features to be activated: `MediaDevices`, `MediaTrackSupportedConstraints`*"]
    pub fn get_supported_constraints(this: &MediaDevices) -> MediaTrackSupportedConstraints;
    # [wasm_bindgen (catch , method , structural , js_class = "MediaDevices" , js_name = getUserMedia)]
    #[doc = "The `getUserMedia()` method."]
    #[doc = ""]
    #[doc = "[MDN Documentation](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia)"]
    #[doc = ""]
    #[doc = "*This API requires the following crate features to be activated: `MediaDevices`*"]
    pub fn get_user_media(this: &MediaDevices) -> Result<::js_sys::Promise, JsValue>;
    #[cfg(feature = "MediaStreamConstraints")]
    # [wasm_bindgen (catch , method , structural , js_class = "MediaDevices" , js_name = getUserMedia)]
    #[doc = "The `getUserMedia()` method."]
    #[doc = ""]
    #[doc = "[MDN Documentation](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia)"]
    #[doc = ""]
    #[doc = "*This API requires the following crate features to be activated: `MediaDevices`, `MediaStreamConstraints`*"]
    pub fn get_user_media_with_constraints(
        this: &MediaDevices,
        constraints: &MediaStreamConstraints,
    ) -> Result<::js_sys::Promise, JsValue>;
}

I wrote a test in /web-sys/tests :

Now when I follow the instruction to run my test this way : https://rustwasm.github.io/docs/wasm-bindgen/contributing/web-sys/testing.html

My computer gets out of Memory Ram, and rustc takes more than 4 Giga of it. I can see Rustc usage of memory going up very quickly and freezing my computer. ( only if I am inside web_sys )

So I use this command, but it said there is no test to run on root of wasm-bindgen

cargo test test_get_display --target wasm32-unknown-unknown --all-features

=>

test result: ok. 0 passed; 0 failed; 172 ignored


use wasm_bindgen_test::*;
use web_sys::{HtmlMediaElement, MediaStream, MediaStreamConstraints, window};
use wasm_bindgen::JsValue;

#[wasm_bindgen_test]
fn test_get_display() {

    let mut constraints = MediaStreamConstraints::new();
    constraints.video(&JsValue::from(true));
    let media_stream_promise : Result<MediaStream, JsValue> = window()
        .navigator()
        .media_devices()
        .unwrap()
        .get_display_media(constraints);

   assert_eq!(media_stream_promise.is_ok(),true);
}

Is there something I am missing ?

I did follow the instruction there https://rustwasm.github.io/docs/wasm-bindgen/contributing/web-sys/testing.html

arn-the-long-beard commented 3 years ago

Thank you @MartinKavik for the help :wink:

We can now share a screen and it works very nicely :smile: Should we close this issue maybe?

I also have a new question now :stuck_out_tongue: Trying to come from this javascript example to get a screenshot with it https://hackernoon.com/how-to-take-screenshots-in-the-browser-using-javascript-l92k3xq7 but I am failing to get the canvas drawing the image.

Should I open a new issue to ask more information about draw_image_with_html_video_element or do we have example somewhere ?

rustwasm / wasm-bindgen

Best way to access unsupported WD API's from web_sys? #2136

Summary

Additional Details