markafoltz
commented
1 month ago Ideally all of these effects would be applied on a per-track basis so that users are free to use their camera with multiple applications at once with their preferred effects.
The reality is that if the OS is applying the effects, then browsers (including Chrome) are unable to provide a stream both with and without effects.
If the browser is applying the effects, then this is something that is more manageable, although there are implementation and performance challenges. For example if you have three effects there are 8 possible stream configurations, and the system may not be able to provide all of them simultaneously because of resource limitations (CPU/GPU/NPU/memory) or architectural limitations.
Implementation wise, today the blur behind the flag (--enable-blink-features=MediaCaptureCameraControls) does not allow two different MediaStreamTrack with different behaviour (one with blur and another without blur), because we are exposing the platform / global behaviour, which means we cannot have an "unprocessed and processed" frame simultaneously for Blur, face framing, eye gaze correction. Only exception is Background Segmentation Mask, details here.
More of an implementation question, does this new proposal allow two different MediaStreamTrack with different behaviour ? Is this something which is going to be exposed as a choice to user as an additional UI permission ? Is this generally needed for Face framing, eye gaze correction, etc.? Implementation wise, all ImageCapture features work similar, where 2 MST do not have different behaviour.