search

mozilla / standards-positions

https://mozilla.github.io/standards-positions/
Mozilla Public License 2.0
611 stars 69 forks source link

Face Detection MediaStreamTrack constraints and VideoFrame metadata #706

Open ttoivone opened 1 year ago

ttoivone commented 1 year ago

Request for Mozilla Position on an Emerging Web Specification

Other information

Updated PR based on feedback and the new WebCodecs VideoFrameMetadata dictionary is here: mediacapture-extensions #78.

zcorpan commented 1 year ago

cc @jan-ivar

jan-ivar commented 1 month ago

Sorry for the late response. I don't have a strong opinion here.

This is not facial recognition, rather it detects where faces are (eyes, mouth). Its purpose would be optimizing quality and power consumption of video effects like background replacement and other augmentation popular in video conferencing.

This seems somewhat useful in providing parity with non-web applications in providing hardware assistance (from drivers/OS etc) with detecting outlines of heads in camera video, where available.

Privacy-wise the segment information does not appear to reveal anything about a videoframe that couldn't be deduced through processing by the application itself. It might reveal specifics about a particular camera driver or OS, but this seems minor compared to other information exposed by an active camera and its device label.

It would only benefit applications that are aware of and utilize this API. I don't have any info on how prevalent hardware is, nor existing usage in other browsers. Absent that, a criticism might be why this functionality is tied to a camera source and that time might be better spent on a broader API that can process web-provided frames as well, like the one mentioned in the explainer. But if that's where the technology exists, that's where it is.