Captured Mouse Events

[eladalon1983]

Hej!

I'm requesting a TAG review of Captured Mouse Events.

We expose an oncapturedmousechange EventHandler on CaptureController. Events are of the general shape {surfaceX: long, surfaceY: long}, exposing the coordinates of the mouse relative to the origin of the captured surface.

• Explainer: https://github.com/screen-share/captured-mouse-events/blob/main/README.md
• Specification URL: https://screen-share.github.io/captured-mouse-events/
• Tests: https://github.com/web-platform-tests/wpt/tree/master/captured-mouse-events
• Security and Privacy self-review: https://github.com/screen-share/captured-mouse-events/blob/main/security-privacy-questionnaire.md
• GitHub repo: https://github.com/screen-share/captured-mouse-events
• Primary contacts (and their relationship to the specification):
  • Elad Alon (@eladalon1983, Google): Spec editor, reviews Chromium implementation
  • Frédéric Wang (@fred-wang, Igalia): Spec contributor, implements in Chromium
• Organization(s)/project(s) driving the specification:
  • Google
  • Igalia
• Key pieces of existing multi-stakeholder review or discussion of this specification:
  • Minutes of SCCG meeting #1
  • Minutes of SCCG meeting #2
• External status/issue trackers for this specification (publicly visible, e.g. Chrome Status):
  • ChromeStatus entry

Further details:

• [X] I have reviewed the TAG's Web Platform Design Principles
• The group where the work on this specification is currently being done: Screen Capture CG
• The group where standardization of this work is intended to be done (if current group is a community group or other incubation venue): Screen Capture CG
• Major unresolved issues with or opposition to this specification: N/A
• This work is being funded by: Google, Tella

We'd prefer the TAG provide feedback as: 🐛 open issues in our GitHub repo for each point of feedback

[atanassov]

Hi @eladalon1983, looking at the proposal with @hober during our Aug 23 f2f, we have the following questions:

1. What is the expected behavior for OSs that have no cursor - iOS, Android etc.?
2. The intended effect assumes that the receiving UA will have the ability to change the lower layer OS cursors effects - is this possible? (For instance, we suspect this would be very diffcult on Windows.)
3. It is unclear how this feature will be available in different pixel densities, DPI, frame sizes etc.
4. What other alternatives have you considered?
5. How does this featuere interact with the a11y stack given they usually have thier own virutal cursor?
6. Not a question but suggestion - we assumed that the spec is your explainer given the little information provided in the README that is linked as an explainer. Can you consolidate please?

[eladalon1983]

1. What is the expected behavior for OSs that have no cursor - iOS, Android etc.?

Invisible "pointers" (touch, stylus) present a different security/privacy profile, because (a) they were not previously exposed to the capturing application, and (b) they involve exposing additional information that is similar to mouse-clicks. As such, they are outscoped, and we don't have concrete plans to extend the API to support them. (However, it seems like it will be possible if desired in the future, although likely subject to some additional permissions prompt.)

2. The intended effect assumes that the receiving UA will have the ability to change the lower layer OS cursors effects - is this possible? (For instance, we suspect this would be very diffcult on Windows.)

Changing the lower layer OS cursor is NOT required for any of the use-cases discussed. Rather, it is assumed that it is possible to exclude the OS cursor from capture. It is then straightforward to draw a custom cursor into the video that is either transmitted remotely or saved to disk. The local user would sitll see their own natural cursor locally; augementation for the local user is not part of the use-case discussed.

3. It is unclear how this feature will be available in different pixel densities, DPI, frame sizes etc.

Could you please help me understand the challenge here? Is it that the specification does not currently clarify whether the coordinates are used in terms of the captured frame's "pixel-space", running from (0, 0) to (max_x, max_y) as defined by MediaStreamTrack.getSettings().width and MediaStreamTrack.getSettings().height?

4. What other alternatives have you considered?

Exposure on the frames themselves rather than using dedicated events. This alternative is seen as complementary, and might be pursued in the future. The downside is that the frequency of the video might be quite low. For example, assume you want to capture at 5fps but render a smooth-looking video remotely by updaitng the cursor at 30fps.

5. How does this featuere interact with the a11y stack given they usually have thier own virutal cursor?

I think it should be up to the UA to decide whether, on the given platform with its particular implementation-limitations for the specific virtual cursor, the virtual cursor is treated as the real cursor, or is ignored.

6. Not a question but suggestion - we assumed that the spec is your explainer given the little information provided in the README that is linked as an explainer. Can you consolidate please?

If @fred-wang does not pick it up first, I can do so when I have more time to devote to this project.

[rhiaro]

Hi, have you had any time to make an explainer which is more in line with our explainer guidance?

[hober]

1. What is the expected behavior for OSs that have no cursor - iOS, Android etc.?

Invisible "pointers" (touch, stylus) present a different security/privacy profile, because (a) they were not previously exposed to the capturing application, and (b) they involve exposing additional information that is similar to mouse-clicks. As such, they are outscoped, and we don't have concrete plans to extend the API to support them. (However, it seems like it will be possible if desired in the future, although likely subject to some additional permissions prompt.)

Instead of outscoping & having bare surfaceX/Y attributes on the event, why not have single attribute that contains an array of x,y pairs? That way you don't have to do any work up-front for any of the N != 1 cases, but you're future-proofed for those cases if they do arise.

[matatk]

This spec presents a great opportunity for improving accessibility (as is alluded to in the text of the spec) - it would be great to highlight this by way of including an "Accessibility considerations" section, where this can be explicitly discussed. Also it would be a good idea to get an accessibility review from the Accessible Platform Architectures (APA) WG, who could help with drafting the suggested accessibility considerations section.

[torgo]

As we haven't heard back from the requestor for some time on this I suggest we may want to close as "timed out" this week?

[torgo]

Just noting that the spec hasn't been updated since July of last year and we have not heard back on any of our questions. Also there has been no action on either the WebKit or Mozilla standards position threads. Hence we are going to go ahead and close this as timed out for now. Please ping us if there is any update.