EyeGazeCorrection.

[riju]

Why ?

We have all heard “eyes are the window to the soul” and their importance in effective communication. The disparity of locations of the subject and the camera make it hard to have eye contact during the video call. Recent consumer-level platforms have been able to solve the eye gaze correction problem, more often employing custom AI accelerators on the client platforms. The ability to render the gaze corrected face would help in a realistic imitation of real-world communication in an increasingly virtual world and undoubtedly be a welcome feature for the WebRTC developer community, something native platforms have been offering for some time.

Microsoft eloquently blogged about EyeGazeCorrection for their Surface lineup. MediaFoundation already has a KSCAMERA_EXTENDEDPROP_EYEGAZECORRECTION_ON starting from Windows 11, provided there is driver support.

Apple’s FaceTime already has something very similar in the form of Attention Correction on iOS 14.0 or later devices.

How ?

Strawman Proposal

<script>
const videoStream = await navigator.mediaDevices.getUserMedia({
   video: true,
});

// Show camera video stream to the user.
const video = document.querySelector("video");
video.srcObject = videoStream;

// Get video track capabilities.
const videoTrack = videoStream.getVideoTracks()[0];
const capabilities = videoTrack.getCapabilities();

// Check whether eyegazeCorrection is supported.
if (!capabilities.eyegazeCorrection) return;

async function applyEyegazeCorrection() {
  try {
    await track.applyConstraints({
      eyegazeCorrection: true;
    });
  } catch (err) {
    console.error(err);
  }
}
</script>

[alvestrand]

1) Why is this filed in the mediacapture-image repo, which is concerned with still images?

2) I had another conversation about eye gaze correction this week, where it was stated that many attempts at this have slipped into the "uncanny valley" where attempts at eye gaze correction have given results that are unsettling due to looking almost natural, but people still see that "something is wrong". Are we sure that this is worth pursuing?

[riju]

1. I should have filed on the main Media Capture and Streams repo or somewhere else ..

2. Okay, if your team has conducted user studies with evidence of people feeling "something wrong", we can put this on the back burner and re-visit later when users are more comfortable with this. I thought that if Microsoft and Apple had launched this feature on their native platform, there must be a user need. Do let us know if subsequent user studies suggest the perception of "uncanny valley" among majority of users has changed.

[youennf]

• I should have filed on the main Media Capture and Streams repo or somewhere else ..

https://github.com/w3c/mediacapture-extensions is probably the way to go nowadays