Captured Surface Control was discussed in the May 2024 WebRTC WG Virtual Interim meeting.
The API allows an application to forward wheel events to and change the zoom level of a surface captured by getDisplayMedia().
The conclusion was to continue the discussion here. Some take-aways for the discussion:
The use case of the local user scrolling and zooming in/out the captured tab is appealing and worth exploring.
The use case of a remote user controlling the tab should not be supported as it opens the door to privacy concerns (e.g., a remote user scrolling the local user's photos)
This API should not be extended to support keystrokes, mouse presses or any other similar user-input actions.
Is the UA in a better position to do these actions?
Arguments in favor of letting the browser do the actions:
Better privacy
By definition ensures the remote control use case is impossible
Arguments in favor of letting the application do the actions:
Better user experience (e.g., better control of the UI and how actions work). This is especially important for zooming, which requires visible UI controls;, but also for scrolling (for example, if processing the capture track with mediacapture-transform, if there are transparent occluding elements, etc.).
The remote control use case can be made impossible in practice via restrictions on the APIs (e.g., requiring user activation for zoom, and local scrolling for wheel events)
Note: Since wheel events don't require dedicated UI controls, they could be handled by the browser, with a simpler API to enable/disable the forwarding. Still, allowing the application to do it
Captured Surface Control was discussed in the May 2024 WebRTC WG Virtual Interim meeting.
The API allows an application to forward wheel events to and change the zoom level of a surface captured by
getDisplayMedia()
.The conclusion was to continue the discussion here. Some take-aways for the discussion:
Arguments in favor of letting the browser do the actions:
Arguments in favor of letting the application do the actions:
cc @youennf, @jan-ivar