twilio / twilio-video-app-react

A collaboration application built with the twilio-video.js SDK and React.js
Apache License 2.0
1.8k stars 725 forks source link

Add 3D Face Effects Twilio Video Processor and Menu #768

Open eludadev opened 1 year ago

eludadev commented 1 year ago

Contributing to Twilio

All third party contributors acknowledge that any contributions they provide will be made under the same open source license that the open source project is provided under.

Pull Request Details

Description

This PR adds native support for 3D face mask effects to this app. User can toggle a selection menu (like background effects) to pick from a list of face masks, and apply the chosen mask to cover their face. It uses WebGL for rendering the mask texture into the user's predicted face geometry.

The model used is TensorFlow's Face Landmarks model. The WebGL code is inspired by Yoshiteru’s implementation, which is under an MIT license and all credits were given. Their code was modified to take advantage of TypeScript and OOP.

Burndown

Before review

Before merge

eludadev commented 1 year ago

https://user-images.githubusercontent.com/101929811/202291149-f1036f96-ea00-48db-ad34-8e90f0270033.mp4