Region Capture

[eladalon1983]

Braw mornin' TAG!

I'm requesting a TAG review of Region Capture.

An API for cropping a video track derived from display-capture of the current browsing context.

• Explainer: https://github.com/w3c/mediacapture-region/blob/main/README.md
• Specification URL: https://w3c.github.io/mediacapture-region/
• Security and Privacy self-review: https://github.com/w3c/mediacapture-region/blob/main/Questionnaire.md
• GitHub repo (if you prefer feedback filed there): https://github.com/w3c/mediacapture-region/
• Primary contacts (and their relationship to the specification):
  • Elad Alon (@eladalon1983), Google
• Organization(s)/project(s) driving the specification: Google
• Key pieces of existing multi-stakeholder review or discussion of this specification:
• External status/issue trackers for this specification (publicly visible, e.g. Chrome Status):
  • Chrome Status entry

Further details:

• [X] I have reviewed the TAG's Web Platform Design Principles
• Relevant time constraints or deadlines: N/A
• The group where the work on this specification is currently being done: WebRTC WG
• Major unresolved issues with or opposition to this specification: N/A
• This work is being funded by: Google

We'd prefer the TAG provide feedback as: 🐛 open issues in our GitHub repo for each point of feedback

[cynthia]

First of all, this looks pretty useful.

API surface-wise, what happens when one tries to create a crop target on a element that is not attached to a document, hence not visible? (I assume it can be rendered and therefore can be captured, but seems like a bug or an interesting side-effect feature?)

I see benefits from having the element being the region boundary, but also a bit unconventional. Initially we thought this was replicating what native does and provides a bounding box to be streamed, but that does not seem to be the case here. This feels like a user expectation that should be fulfilled.

We weren't able to figure out the user flow of how the preview is presented the user in the example code. Is it triggered when the crop target promise is being returned? (Presumable rejecting if the user dismisses the preview as "please don't"?)

[eladalon1983]

First of all, this looks pretty useful.

Thanks. :-)

API surface-wise, what happens when one tries to create a crop target on a element that is not attached to a document, hence not visible? (I assume it can be rendered and therefore can be captured, but seems like a bug or an interesting side-effect feature?)

The thing that happens if one creates a CropTarget for an element that is attached, but then detaches the element. Namely, the track is muted¹.

Initially we thought this was replicating what native does and provides a bounding box to be streamed, but that does not seem to be the case here.

I think you initial interpretation was in fact correct. The element defines a bounding box to be streamed.

This feels like a user expectation that should be fulfilled.

The user is not in control of the feature; the application is. The application could be saving the cropped file to disk, storing it remotely, doing OCR inside of it... Anything, really. Whether it's user-facing or not is up to the application, and not up to the API.

We weren't able to figure out the user flow of how the preview is presented the user in the example code.

Could you please clarify the question? What preview are you asking about? (Note clarification above.)

-- [1] The muting part is currently under discussion. It's unclear at the moment if we'll set MediaStreamTrack.muted, but regardless, no frames will be delivered, so "effectively muted."

[cynthia]

Thank you for the clarification!

I think you initial interpretation was in fact correct. The element defines a bounding box to be streamed.

I think there is a minor misunderstanding here, we understood this as a bounding box (draw a rectangle) in any part of the desktop - which is not what this proposal is about.

Given that the naming/expectations might introduce confusion - may we suggest renaming this to something that suggests a specific scope? (e.g. "Tab" Region Capture)

Additionally, we'd like the group to consider the unfulfilled user need of "stream arbitrary region from any part of the screen" in WebRTC at some point.

Thank you for bringing this to our attention, and we are happy to see this proposal move forward.