Broader API proposals for device selection and permissions

[hills]

As supplement to a previous ticket I posted a broader API suggestions, and it was suggested these would be a good 'issue' in their own right for discussion.

I've tried to wholistically address a number of issues around permissions, fingerprinting and device selection.

I'm afraid I'm very much just a user of these APIs on a practical level, with a day job, so that doesn't leave me a lot of time to comb the specifications or be intricately in tune with details of every discussion. Though I have been reading here to try and get up to speed with the issues to do my best to ensure this is relevant.

I do hope you will see this from a distance; reining back complexity and special cases, not adding more, and within the constraints of the existing API. I hope you don't mind considering broader goals from an outside contributor.

---

Calls to getUserMedia that do not specify a device ID (or specify "default") would be governed by a "permission to use your camera/microphone" dialogue provided by the browser:

• The browser should provide option to choose a device here
• The persistence of this permission does not need to be spec'd
  • Browser or user policy can device: eg. forever; this session
  • Or every call to getUserMedia()
  • Spec may demand some implicitly permitted operations; eg. if the device is already open by that page
• This means that most web developers can just call getUserMedia, once, and never worry about enumerateDevices

And then, independently a permissions flag (looks like [[canExposeDeviceInfo]]?):

• Browser prompts for "permission to use a range of media devices on your system"
• Governs access to anything involving a deviceID: eg.
  • enumerateDevices()
  • getUserMedia use with a deviceID
  • setSinkID
  • "new device available" event
• Pop up the permissions dialogue on the first of any of these API uses
  • Even if the page remembers a deviceID in a cookie, it must still have this permission to make an API call with it
• Browser policy or user sets how long this permission is granted for: eg. forever; this session; next 5 minutes
• This dialogue can provide access control
  • eg. "default devices only", "microphones only", or a specific allowlist
• These deviceIDs now act as a pass on getUserMedia(), and not subject to the checks above.

And that's it!

What my goals are in the above proposal:

• Clarity of not having permissions based on events ordering:
  • enumerateDevices() needing to happen after getUserMedia()
  • 'fake' data now, complete data later should be considered evil
• Tackle fingerprinting issues; no deviceIDs are ever seen by a page without user permission
• User is clear what they granted permission on
  • No opaque relationships
  • which page can access which devices, not implicit based on history (#703)
• Retain compatibility with exisiting code
• Prevent 99% of developers having to build device selection UIs unnecessarily:
  • Chromium presently omits device selection on getUserMedia, which means most developers need to call enumerateDevices at present to give good user experience, and do this after an early getUserMedia() to call for permissions.
• Leave policy to the browser
  • Spec can be simpler and less bug footprint
  • Policy is variable in different browsing environment; eg. mobile vs. desktop vs. pro user/producitivity app
  • Policy is going to need to evolve without breaking existing sites or forcing them to new APIs

One thing the above embodies, which seems to be fighting in the other issues (such as w3c/mediacapture-extensions#6) is whether deviceID is a first class consideration. Whilst its clever to try and channel deviceID as just another constraint, the separation of 'what' media the client has access to, versus 'how' it samples that media, could be inevitible.

It is good to remember that not all apps are standard video conferencing apps, and already WebAudio apps for producivity use multiple devices concurrently.

[youennf]

• enumerateDevices() needing to happen after getUserMedia()

You can call enumerateDevices before getUserMedia. It will tell you whether there are microphones and/or cameras, which should give you enough to craft your UI and getUserMedia request.

• 'fake' data now, complete data later should be considered evil

This is not fake data, this is limited data and complete data if user agrees.

• Tackle fingerprinting issues; no deviceIDs are ever seen by a page without user permission

The current design enforces this.

• Prevent 99% of developers having to build device selection UIs unnecessarily:

There is a proposal for a device picker that hopefully can address that.

whilst its clever to try and channel deviceID as just another constraint, the separation of 'what' media the client has access to, versus 'how' it samples that media, could be inevitible.

I agree. getUserMedia model is made more complex than it should because we try to mix selection and setup in one step. Selection should be the primary focus of getUserMedia, setup can be mostly delegated to applyConstraints.

To make progress here, it would be nice to understand the exact cases you find difficult or impossible to implement with the current API design. This would make it much easier to try updating the current design.

[guest271314]

There is a proposal for a device picker that hopefully can address that.

Can you kindly link to that proposal?

Selection should be the primary focus of getUserMedia

That is currently impossible using initial call to getUserMedia() alone at Chromium due to

Chromium presently omits device selection on getUserMedia