Auto-pause capture when user switches captured content

[eladalon1983]

Both Chrome and Safari now allow the user to change what surface is captured.

That's obviously great stuff. Can we make it better still?

• What if an application intends to do some processing that depends on the captured content?
• What if an application wants to set different constraints, for instance when capturing a window vs. when capturing a screen?
• What if the application intends to save different surfaces to different files, and wants to start appending to a new file whenever the user changes the source?

So I propose that we add two things:

1. A control which allows an application to instruct the browser - whenever the user changes what surface is shared, pause the track (by setting enabled to false).
2. Fire an event whenever this happens. (With the general expectation that the application will apply new processing and then set enabled back to true.)

Possibly the two can be combined, by specifying that setting an event handler signals that the pausing behavior is desired (@alvestrand's idea).

Another natural extension of this idea is to also apply it when a captured tab undergoes cross-origin navigation of the top-level document. When that happens, some applications might wish to stop capture momentarily and (in-content) prompt the user - "do you still want to keep sharing?"

Relevant previous discussion here.

[youennf]

• Cross-type surface-switching

ok

• Add audio-sharing after the fact

Is this already existing? This seems like a difficult topic: multiple UX possibilities with different user intents, interaction with permissions, removal of audio sharing after the fact without removing display sharing... This might deserve its own issue, though it is fine to keep it here as one of the longer term issue we want to solve in that space.

• Frame delivery is clearly separated for the captured surfaces before and after the switch.

The first two points are user driven, this last one is already phrased in terms of API. The question is what we are trying to achieve with this for the user. How about:

• Allow web applications to control whether and where sending video frames from a newly captured surface.

We have a solution when starting to capture and we want a solution when user decides to switch surface, whether same type or different type.

[tovepet]

After the discussions in the Captured Surface Switching - Working Doc, it looks like the most promising way to reach agreement is to continue with the existing injection model and add the following extensions: auto-pause to allow applications time to set up the capture of a new surface and avoid transmitting frames associated with the wrong capture. allowCrossTypeSurfaceSwitching option to enable cross-type surface-switching alwaysReturnAudio option for getDisplayMedia to always return an audio track if audio is requested, so that audio be muted/unmuted by the user at their leisure. Together, these three extensions should cover most of the use-cases we have discussed in this issue.

I have created three new issues for these extensions so that we can tackle them individually: auto-pause: https://github.com/w3c/mediacapture-screen-share-extensions/issues/15 allowCrossTypeSurfaceSwitching: https://github.com/w3c/mediacapture-screen-share-extensions/issues/16 alwaysReturnAudio: https://github.com/w3c/mediacapture-screen-share-extensions/issues/17

[eladalon1983]

Burn the ships.