Spec should be more explicit about exposure of deviceId

w3c / mediacapture-screen-share

Media Capture Screen Capture specification

https://w3c.github.io/mediacapture-screen-share/

Other

86 stars 28 forks source link

Spec should be more explicit about exposure of deviceId #306

Open eladalon1983 opened 2 weeks ago

eladalon1983 commented 2 weeks ago

The spec says:

Display capture sources therefore cannot be selected with the deviceId constraint, since their deviceIds are not exposed.

This implies, weakly, that the MediaTrackSettings dictionary associated with screen-sharing tracks should not contain an entry whose key is "deviceId". It would be better to mention this explicitly.

youennf commented 2 weeks ago

Isn't deviceId somewhat useful? If getDisplayMedia is called twice and user picks the same surface twice, the web page can easily notice it with deviceId. Or let's say user switches from surface A then to surface B then back to surface A, deviceId could potentially allow the web page to know about this.

Given Chrome and Safari are exposing it, maybe the spec could be clarified to allow/mandate it?

eladalon1983 commented 2 weeks ago

If getDisplayMedia is called twice and user picks the same surface twice, the web page can easily notice it with deviceId.

As you point out later in your message, that too would require amending the spec. I'm open to the idea. Some challenges are:

It's non-trivial to specify the "same surface". When is a tab still the same tab? Does navigation change things? Does it matter if it's cross-origin or not? Does reopening a closed tab change the surface?
You suggest that this deviceId should stay the same across capture-sessions (invocations of gDM). But is that only within a given "session" of the capturing application? What if it's reloaded? Unloaded (closed tab) then loaded again 2 hours later? Does it matter if BFCache is used or not? What happens if the user agent is restarted? What if the machine is restarted?

youennf commented 2 weeks ago

For windows and screens, I would guess the OS is providing some IDs to each surface that would be hashed like for cameras. Stability of the IDs would be based on what the OS decides + the hash that might change if cookies are removed. For tabs, it would be up to the UA to decide to allocate IDs to each one of its tab, and then hash them as well.

eladalon1983 commented 2 weeks ago

Windows and screens: You are discussing feasibility, but I am still at the stage where I am trying to understand what behavior we want to see. That is, the OS providing stable IDs means the browser can provide stable deviceIds. But should it?

Tabs: What behavior do we want to specify? If we truly think it's desirable to have deviceIds, we need to specify how long they are stable, to ensure consistent behavior across user agents.

Another issue is dynamic switching. A deviceId tied to a specific surface, will become misleading if we surface-switch. For that reason, I think it's the wrong mechanism for the problem stated ("user picks the same surface twice").

youennf commented 6 days ago

What behavior do we want to specify?

We have the notion of tabs in the HTML spec (I think this is html navigable). deviceId lifetime would be tied to navigable lifetime.

A deviceId tied to a specific surface, will become misleading if we surface-switch.

How so, if deviceId actually changes when calling getSettings() after a surface switch?

eladalon1983 commented 6 days ago

We have the notion of tabs in the HTML spec (I think this is html navigable). deviceId lifetime would be tied to navigable lifetime.

Since you are more well-versed in this than me, could you please help me out by explaining if Navigables or Traversable navigables are the spec entities that correspond to tabs? (And what does the other one generally correspond to?)

How so, if deviceId actually changes when calling getSettings() after a surface switch?

Are deviceIds mutable?

jan-ivar commented 6 days ago

This spec explicitly specifies which constraints to implement, as required to by:

§ 17.4 Defining a new source of MediaStreamTrack which says: "Other specs ... will need to declare which constrainable properties..., if any, are applicable to each kind of media this new source produces, and how they work with this source"

They're listed under § 5.4 Constrainable Properties for Captured Display Surfaces. deviceId is not on the list and therefore not specified.

Based on this I suggest closing this issue.

jan-ivar commented 6 days ago

"constraints defined for getUserMedia() do not apply unless listed here."

youennf commented 6 days ago

Well, two implementations are exposing deviceId. And it might be useful as part of getDisplayMedia or as part of source switching. I would change the spec to expose deviceId if we deem it useful enough.