search

w3c / mediacapture-extensions

Extensions to Media Capture and Streams by the WebRTC Working Group
https://w3c.github.io/mediacapture-extensions/
Other
19 stars 15 forks source link

Simplify device selection algorithm #18

Open youennf opened 3 years ago

youennf commented 3 years ago

Current approach suggests to test all possible constraint combinations and use the combination that has the lowest fitness distance. This is difficult to implement and provides sometimes interesting but maybe unexpected results. It would be good to try providing a simpler algorithm which could hopefully be implemented consistently.

youennf commented 3 years ago

As an example: getUserMedia({ deviceId: 'abcdef', width: 1920, height: 1080, frameRate: 60 }) If device 'abcdef' is there and is not HD, it may or may not be selected depending on whether there is a HD camera.

A simpler algorithm would be to try doing the search on one constraint at a time, following always the same order. For instance deviceId > groupId > facingMode > width ... In the above example, if 'abcdef' is found, it would always be the one that is selected.

One way to implement this with the existing algorithm is to provide a weight to each constraint. If deviceId has a weight of 32, groupId 16, facingMode 8...

alvestrand commented 3 years ago

Comment w3c/mediacapture-main#1 and comment w3c/mediacapture-main#2 address completely different issues. "Try all possible combinations" is a description of what result we want, not an implementation prescription; the more efficient implementation for numeric constraints is to treat it as a ~linear programming problem (defining the part of the search space that can contain a solution, and seeing if existing solutions are in that part of the solution space). We do have an open source implementation of this.

The WG long ago decided to not prescribe exactly which permissible device to return when more than one was permissible (since the example you give has no exact constraints, all devices will be permissible).

The lack of an API for generating fake devices makes it very difficult to test this stuff in Web Platform Tests, but I think you're working on a solution for that, right?

Re comment w3c/mediacapture-main#2, trying one constraint at a time: If you want this algorithm, you write it as {deviceid: 'abcdef', advanced: { width: 1920, height: 1080, framerate: 60}} - embedding a specific choice of priority into the browser when we don't have to seems to me like a wrong decision.

guest271314 commented 3 years ago

The lack of an API for generating fake devices makes it very difficult to test this stuff in Web Platform Tests, but I think you're working on a solution for that, right?

let dev = navigator.mediaDevices.createDevices(bring_and_or_dynamically_create_your_own_devices)

will be helpful. Where to track development of that feature?