react-native-deepar

Snapchat-like filters, AR lenses, and real-time facial animations.

React-Native wrapper for DeepAR.

Table of Contents

• What is DeepAR?
• Compatibility
• Getting Started
  • Other Required Steps
  • Android
    • Proguard
  • iOS
  • Requesting Permissions
  • Getting Permissions
  • Requesting Permissions
• Installing AR Models
• Using AR Models over Internet
• Usage
• API
  • Props
  • Events
  • Methods
  • General
  • Camera Control
  • Video Recording
  • Change Parameters
  • Core
  • Camera Module
• Background Segmentation
• Face Painting
• Watermark
• Example App
• Limitations
• Contributing
• License

What is DeepAR?

Preview
This GIF taken from DeepAR offical site.

DeepAR is an infrastructure where you can make AR applications in ease. DeepAR is not free, but you can create applications that can be used by up to 10 people for testing purposes for free.

• Check pricing from Pricing Page.
• In order to use DeepAR, you need to generate an API key. You can generate the API key from the Developer Panel.
• With the Asset Store, you can buy ready to use AR content.
• If you're looking for free filters, here is some: Free Filter Pack
• With the DeepAR Studio, you can create, edit and fine tune your own AR content.
• To learn DeepAR Studio, visit DeepAR Help Center.

DeepAR Features:

• Face Filters, Effects and Masks
• AR Beauty & Makeup
• Realtime Hair Color Segmentation
• Background Removal and Segmentation
• Realtime Emotion Detection

You can visit DeepAR's offical site to learn more.

Compatibility

Getting Started

:warning: This library under development, if you found a bug, please open an issue from here.

:warning: It only works on physical devices, not will work with simulator.

yarn add react-native-deepar

Other Required Steps

Android

1. Open your project's AndroidManifest.xml and add the following lines:

<uses-permission android:name="android.permission.CAMERA" />

<!-- optionally, if you want to record audio: -->
<uses-permission android:name="android.permission.RECORD_AUDIO" />

2. Update your minSdkVersion version minimum 23, compileSdkVersion and targetSdkVersion version minimum 31 from android/build.gradle file, like below:

buildscript {
    ext {
        buildToolsVersion = "29.0.3"
-        minSdkVersion = 20
-        compileSdkVersion = 30
-        targetSdkVersion = 30
+        minSdkVersion = 23
+        compileSdkVersion = 31
+        targetSdkVersion = 31
    }
}

Proguard

If you're using Proguard, make sure to add rules below:

-keepclassmembers class ai.deepar.ar.DeepAR { *; }
-keepclassmembers class ai.deepar.ar.core.videotexture.VideoTextureAndroidJava { *; }
-keep class ai.deepar.ar.core.videotexture.VideoTextureAndroidJava

iOS

1. Open your project's Info.plist and add the following lines:

<key>NSCameraUsageDescription</key>
<string>$(PRODUCT_NAME) needs access to your Camera.</string>

<!-- optionally, if you want to record audio: -->
<key>NSMicrophoneUsageDescription</key>
<string>$(PRODUCT_NAME) needs access to your Microphone.</string>

2. Open your ios/YourProject.xcworkspace file in Xcode and update your iOS version to 11.0 minimum, like below:

Setting iOS Version from Xcode
Follow steps in the picture.

3. Add DeepAR.xcframework to Build Phases:

Add DeepAR to Build Phases (1)	Add DeepAR to Build Phases (2)
Follow steps in the picture.	Follow steps in the picture.

Note: Don't forget install Pods for iOS and rebuild your app.

Requesting Permissions

You need to ask necessary permissions before render the DeepAR component.

Getting Permissions

Simply use the get functions to find out if a user has granted or denied permission before:

import { Camera } from 'react-native-deepar';

// ..

const cameraPermission = await Camera.getCameraPermissionStatus();
const microphonePermission = await Camera.getMicrophonePermissionStatus();

A permission status can have the following values:

• authorized: Your app is authorized to use said permission. Continue with using the <DeepAR> view.
• not-determined: Your app has not yet requested permission from the user. Continue by calling the request functions.
• denied: Your app has already requested permissions from the user, but was explicitly denied. You cannot use the request functions again, but you can use the Linking API to redirect the user to the Settings App where he can manually grant the permission.
• restricted: (iOS only) Your app cannot use the Camera or Microphone because that functionality has been restricted, possibly due to active restrictions such as parental controls being in place.

Requesting Permissions

Use the request functions to prompt the user to give your app permission to use the Camera or Microphone.

import { Camera } from 'react-native-deepar';

// ..

const cameraPermission = await Camera.requestCameraPermission();
const microphonePermission = await Camera.requestMicrophonePermission();

• authorized: Your app is authorized to use said permission. Continue with using the <DeepAR> view.
• denied: The user explicitly denied the permission request alert. You cannot use the request functions again, but you can use the Linking API to redirect the user to the Settings App where he can manually grant the permission.

Installing AR Models

1. Put your AR models into a desired destination
2. Create react-native.config.js like below:

module.exports = {
  assets: ['./assets/effects'], // <-- example destination
};

3. Add a NPM script like below:

{
  "scripts": {
+   "asset": "./node_modules/.bin/react-native-copy-asset"
  }
}

4. Run the script to link your AR models

npm run asset

Note: If you remove an AR model, you can run the same command for unlinking removed asset.

Using AR Models over Internet

You don't have to install AR models in your app, you can use AR models over Internet.

1. Install rn-fetch-blob to your project
2. You can use AR models over internet like below:

import RNFetchBlob from 'rn-fetch-blob';

RNFetchBlob.config({
  fileCache: true,
})
  .fetch('GET', 'http://betacoins.magix.net/public/deepar-filters/8bitHearts')
  .then((res) => {
    deepARRef?.current?.switchEffectWithPath({
      path: res.path(),
      slot: 'effect',
    });
  });

Usage

Make registration to DeepAR and get an API key from Developer Panel.

import React, { useRef } from 'react';
import DeepAR, { IDeepARHandle } from 'react-native-deepar';

const App = () => {
  const deepARRef = useRef<IDeepARHandle>(null);

  return (
    <DeepAR
      ref={deepARRef}
      apiKey="your-api-key"
      style={{ flex: 1 }}
      onInitialized={() => {
        // ..
      }}
    />
  );
};

API

The <DeepAR> component can take a number of inputs to customize it as needed. They are outlined below:

Props

Events

These are various events that you can hook into and fire functions on in the component:

Methods

These are the various methods.

General

Camera Control

Video Recording

Change Parameters

For more details about changeParameter API read this article here.

Core

Camera Module

import { Camera } from 'react-native-deepar';

Background Segmentation

DeepAR has Background Segmentation feature, with this feature you can change your background in real-time.

Background Segmentation Preview
This image taken from DeepAR offical site.

There is a filter called Background from Free Filter Pack and you can use this filter.

How change background image?

Switch the Background effect and apply new background image like below:

import { TextureSourceTypes } from 'react-native-deepar';
import RNFetchBlob from 'rn-fetch-blob';

RNFetchBlob.config({})
  .fetch('GET', 'https://random.imagecdn.app/450/800')
  .then((res) => {
    deepARRef?.current?.changeParameterTexture({
      gameObject: 'Background',
      component: 'MeshRenderer',
      parameter: 's_texColor',
      type: TextureSourceTypes.BASE64,
      value: res.base64(),
    });
  });

Face Painting

You can detect touches in your effects if the effect customized for detecting touches. For an example; DeepAR has a filter called Face Painting, with this effect you can paint your face in real-time with touches.

For more details about face painting effect read this article here.

Face Painting Preview
This image taken from DeepAR offical site.

If you want to be able to detect touches on the screen, you need to use the following code:

// If you switch to the face painting effect, use below code to ability detect touches over DeepAR view
deepARRef?.current?.setTouchMode(true);

// If you not use face painting effect, use below code to disable detecting touches over DeepAR view
deepARRef?.current?.setTouchMode(false);

Watermark

Would you like to add watermark to your filter? Follow this tutorial: Placing an image as a part of your filter

Example App

Ball Face	Background Segmentation

# clone the project
git clone https://github.com/ridvanaltun/react-native-deepar.git

# go into the project
cd react-native-deepar

# make project ready
npm run bootstrap

# go into the example
cd example

# copy environment file and set your api keys (bundle id is com.example.reactnativedeepar)
cp .env.example .env

# run for android
npm run android

# or

# run for ios
npm run ios

Limitations

• Background Segmantation filter only works iPhone 7 and newer devices for iOS
• Hair Segmantation filter not supported on Android and only works iPhone 7 and newer devices for iOS

Learn more about limitations from features by platform page.

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

This project exists thanks to all the people who contribute.

License

This project is licensed under the MIT License - see the LICENSE file for details.