search

w3c / mediacapture-extensions

Extensions to Media Capture and Streams by the WebRTC Working Group
https://w3c.github.io/mediacapture-extensions/
Other
19 stars 14 forks source link

Face framing #55

Closed eehakkin closed 1 year ago

eehakkin commented 2 years ago

Face framing allows automatic face framing based on the position of the subject within the field of view to focus on the dominate face in view.

MediaFoundation has added the support for face framing using properties like KSCAMERA_EXTENDEDPROP_DIGITALWINDOW_AUTOFACEFRAMING from Windows 11, if there is support from the underlying driver. By encapsulating the preferably (ASIC) hardware accelerated inference work in the driver and leveraging standard platform APIs, we do not have to re-invent the wheel for every web application.

Please see https://github.com/w3c/mediacapture-extensions/pull/49#issue-1099641483 why a constrainable property is most suitable for the purpose.

Preview | Diff

yuhonglin commented 1 year ago

Hi, we are also interested in supporting this in chromeOS. Is there any existing explainer/spec?

yuhonglin commented 1 year ago

May I ask one question: Is there any signal/status showing the following,

  1. The auto-framing is in progress.
  2. The auto-framing is done.
eehakkin commented 1 year ago

Hi, we are also interested in supporting this in chromeOS. Is there any existing explainer/spec?

There is not explainer but we did create an explainer for background blur (https://github.com/riju/backgroundBlur/blob/main/explainer.md) and face framing is quite similar to that in a sense.

The spec is basically only in the CL i.e. it has not yet been approved by the W3C WebRTC WG.

May I ask one question: Is there any signal/status showing the following,

  1. The auto-framing is in progress.
  2. The auto-framing is done.

There is none.

alvestrand commented 1 year ago

Currently waiting for the metadata format to be landed in the webcodecs metadata registry.

eehakkin commented 1 year ago

Currently waiting for the metadata format to be landed in the webcodecs metadata registry.

That would of course be nice but it is not actually directory needed for automatic face framing because the automatic human face based framing is done by the driver or by the platform which do not need webcodecs metadata.

alvestrand commented 1 year ago

Is face framing intended as a "continuous follow" function?

In one trial I'm aware of, continuous follow framing was near-universally hated by the users, while a button they could hit saying "frame my face now" was popular.

The expected behavior should be clear from the functional description.

(That said, I am not at all averse to adding face framing to the spec even if it's not appropriate for many cases - this is, after all, the extensions spec, where we put stuff in order to let it be tried.)

eehakkin commented 1 year ago

Is face framing intended as a "continuous follow" function?

Yes.

The expected behavior should be clear from the functional description.

I updated descriptions.

(That said, I am not at all averse to adding face framing to the spec even if it's not appropriate for many cases - this is, after all, the extensions spec, where we put stuff in order to let it be tried.)

I also added a note that if a need arises, faceFraming booleans could be replaced with faceFramingMode MeteringMode DOMStrings ("none", "manual", "single-shot", "continuous").

eehakkin commented 1 year ago

I changed the example to use required constraints instead of advanced constraints.

dontcallmedom-bot commented 1 year ago

This issue was mentioned in WEBRTCWG-2023-02-21 (Page 46)