Background Blur: Unprocessed video should be mandatory to support

w3c / mediacapture-extensions

Extensions to Media Capture and Streams by the WebRTC Working Group

https://w3c.github.io/mediacapture-extensions/

Other

19 stars 14 forks source link

Background Blur: Unprocessed video should be mandatory to support #121

Open alvestrand opened 8 months ago

alvestrand commented 8 months ago

In the context of https://w3c.github.io/mediacapture-extensions/#exposing-mediastreamtrack-source-background-blur-support and similar mechanisms for platform effects on video tracks, concern has been raised that this type of functionality can cause deep confusion for users when the same class of effect can be applied both in an application (conferencing systems, for instance) and in the platform - users won't know where to turn it off, and double blur may cause effects that were not as intended.

For other types of effects (face touchups, background replacement, emoji reactions), the cost of being unable to prevent confusing interactions between effects is even bigger.

This concern would be alleviated if all such capabilities were defined to be "MUST support constraint=false" - that is, if the application could always turn them off and get an unprocessed video feed.

huibk commented 8 months ago

Since the introduction of platform effects, video conferencing applications on the web have faced increased negative user feedback such as:

Users think the web app is broken because they are confused where and how background blur is turned on and off
Users are noticing distorted video, such as bad cropping/segmentation or added delay (due to double processing)
Users are noticing an increase in battery consumption (due to double processing)
Video effects being used inappropriately or users are confused why certain effects are present

Since it's not really feasible to align what the app does and the platform does, there will always be inconsistency, conflicts and user confusion between the two. Web apps should be able to build experiences that go beyond what specific platforms are offering and without having to burden users with navigating two sets of controls.

dontcallmedom-bot commented 7 months ago

This issue was discussed in WebRTC November 2023 meeting – 21 November 2023 (Issue #121: Background Blur: Unprocessed video should be mandatory 🎞︎)

alvestrand commented 7 months ago

Discussion about this item (which branched into the general "three thumbs up" problem) seemed to be generally positive towards "it should be possible to get unprocessed", with the main issue being raised being that on some OSes, there are currently no controls exposed to that would allow this.

A suggested resolution was to make it a SHOULD, with a note saying that only in the case of UAs operating on platforms where those controls were not available, the lack of ability to get unprocessed data was acceptable.

youennf commented 7 months ago

only in the case of UAs operating on platforms where those controls were not available, the lack of ability to get unprocessed data was acceptable.

I am not sure we should separate OS and UA here. Some OSes may decide that, if user selected background blur in OS UX, no application will be allowed to opt-out without user interacting with OS UX. A UA may decide to apply the same constraint to any web page it runs, even if OS is not imposing such restriction.

The spec offers the flexibility to all those models, which is good, especially since this is quite new territory that needs to mature.

alvestrand commented 7 months ago

UA is the one we're writing rules for, and where we assume that the UA implementors will be listening to what we write. If we think that it's OK for an UA to refuse to allow an application to prevent background fireworks, floating thumbs-up symbols or background blur, we should say that. But that was not the sense I got from the meeting.

alvestrand commented 7 months ago

an interesting question here is if we should mention (or permit) the possibility of applyConstraints() causing user interaction.

youennf commented 7 months ago

If we think that it's OK for an UA to refuse to allow an application to prevent background fireworks

Such a UA should have good reasons for not allowing this. One reason that was mentioned during the call was privacy. This is a valid concern, at least for some of these effects. User may for instance have agreed to permit camera access with the assumption that only their face or part of the captured data would be shared to the web page.

an interesting question here is if we should mention (or permit) the possibility of applyConstraints() causing user interaction.

I think it is allowed, I have no objection mentioning it.

eladalon1983 commented 7 months ago

I wonder if we branch of the discussion of effects which clearly serve no privacy purpose, and find some way to mandate that the user agent SHOULD/MUST at least allow Web applications to avoid those...?

Or at least avoid such reactions being inserted heuristically - through analysis of the user's video - while still allowing users to force them in through explicit interaction with the OS or hardware.