Support face tracking in live streams

[huningxin]

Use Cases

• Live video feeds would like to identify faces in a picture/video as highly salient areas to e.g. give hints to image or video encoders. (from https://discourse.wicg.io/t/rfc-proposal-for-face-detection-api/1642)
• Head tracking for video or VR (from https://discourse.wicg.io/t/rfc-proposal-for-face-detection-api/1642/6)
• Face tracking based real-time human-computer interaction.

  Platform specific implementation notes

  Android

Camera2 CaptureRequest Event-Driven Pipeline for Face Tracking

iOS

Tracking Faces in Video

Rough sketch of MediaStreamFaceDetector

[Constructor(MediaStream stream, optional FaceDetectorOptions faceDetectorOptions)]
interface MediaStreamFaceDetector: EventTarget {
    readonly        attribute MediaStream       stream;
                    attribute EventHandler      onerror;
                    attribute EventHandler      onfacedetected;
};

[Constructor(DOMString type, FaceDetectionEventInit eventInitDict)]
interface FaceDetectionEvent : Event {
  readonly attribute <sequence<DetectedFace>> faces;
};

dictionary FaceDetectionEventInit : EventInit {
  <sequence<DetectedFace>> faces;
};

Usage

navigator.mediaDevices.getUserMedia(constraints).then(stream => {
  var msfd = new MediaStreamFaceDetector(stream, {fastMode: true, maxDetectedFaces: 1});

  msfd.onfacedetected = function(event) {
    for (const face of event.faces) {
      console.log(`Face ${face.id} detected at (${face.x}, ${face.y}) with size ${face.width}x${face.height}`);
    }
  }
});

[yellowdoge]

This looks cool but is more tracking than detection, which is the concern of this Spec. Perhaps we could throw it back at the wicg discuss as a separate but related idea?

[huningxin]

It makes sense to me. Thanks!