Graphics memory increase and app termination issue within tensorflow.js getBoundingBoxes, estimateFaces method

[adjh54ir]

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): localhost
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Galaxy S21 5G(SM-G991N) or ALL
• TensorFlow.js installed from (npm or script link): yarn 1.22.22
• TensorFlow.js version (use command below): @tensorflow/tfjs(3.7.0)
• Browser version: Android OS 14
• Tensorflow.js Converter Version: 3.7.0

Describe the current behavior

hello.

We are developing React Native-based mobile development using the @tensorflow-models/blazeface:0.1.0 library. This is used within a loop that repeats at specific times, but each time the function is called, a problem occurs where the graphics memory increases rapidly as shown below. Can you please tell me a solution to this?

For reference, there is the same problem of graphics memory rapidly increasing in the @tensorflow-models/face-landmarks-detection:0.0.3 version. Can you tell me a solution to this?

[Profiler measured on Galaxy S21 5G device : @tensorflow-models/blazeface:0.1.0]

[Profiler measured on Galaxy S21 5G device : @tensorflow-models/face-landmarks-detection: 0.0.3]

Describe the expected behavior

"dependencies": {
    "react": "18.2.0",
    "react-native": "0.72.4",
    "@tensorflow-models/blazeface": "^0.1.0",
    "@tensorflow-models/face-landmarks-detection": "0.0.3",
    "@tensorflow/tfjs": "3.7.0",
    "@tensorflow/tfjs-backend-webgl": "3.7.0",
    "@tensorflow/tfjs-converter": "3.7.0",
    "@tensorflow/tfjs-core": "3.7.0",
    "@tensorflow/tfjs-react-native": "0.8.0",
}

The code that is executed is as follows.

1. First, load all models through studyCameraHandler.initModel().
2. The Tensor value is received each time through the onReady property of TensorCamera. : studyCameraHandler.readyTensorCamera()
3. In studyCameraHandler.readyTensorCamera(), this is a scenario where measurements are performed through the model stored in the state and a boundbox is returned.
   • Among these, there is a problem that if blazefaceModel.estimateFaces() is performed repeatedly, the graphics memory increases rapidly and the app is terminated.

import { createRef, memo, RefObject, useEffect, useRef, useState } from "react";
import { CommonType } from "../../types/common/CommonType";
import styles from "./styles/StudyPureScreenStyles"
import { SafeAreaView, ScrollView, StatusBar, View } from "react-native";
import { Camera, CameraType } from "expo-camera";
import { DeviceType } from "types/common/DeviceType";
import DeviceInfoUtil from "common/utils/DeviceInfoUtil";
import { cameraWithTensors } from "@tensorflow/tfjs-react-native";
import * as tf from "@tensorflow/tfjs";
import * as blazeface from '@tensorflow-models/blazeface';
import { FaceDetetorType } from "types/FaceDetectorType";
import '@tensorflow/tfjs-backend-webgl'

// Expo-Camera to TensorCamera
const TensorCamera = cameraWithTensors(Camera)

/**
 * 학습(순공) 화면입니다.
 */
const StudyPureScreen = ({ route, navigation, appState }: CommonType.CommonProps) => {

    const timerRef = useRef<NodeJS.Timeout | null>(null);
    const [isCameraOn, setIsCameraOn] = useState(true);
    const tensorCameraRef: RefObject<any> = createRef<Camera>();
    const TEXTURE_DIMS: DeviceType.TextureDims = DeviceInfoUtil.getTextureDims();

    const [initModel, setInitModel] = useState<FaceDetetorType.InitPureModelState>({
        isLoading: false,
        isTensorReady: false,
        blazefaceModel: null
    });
    useEffect(() => {
        studyCameraHandler.initModel();

        return () => {
            console.log("[+] Clean Up!")
            if (timerRef.current) clearTimeout(timerRef.current);
            if (initModel.blazefaceModel) initModel.blazefaceModel.dispose();
            if (initModel.isTensorReady) tf.dispose();                        
        }
    }, []);

    const studyCameraHandler = (() => {

        return {
            initModel: async () => {
                // [STEP1] Tensor Model Load
                let _tensorReady: boolean = false;
                await tf.setBackend('rn-webgl')
                    .then(() => {
                        console.log("[+] Load Tensorflow.js....");
                        _tensorReady = true;
                    })
                    .catch((error) => { console.error(`[-] Tensorflow Ready Error: ${error}`); });

                let _blazefaceModel: blazeface.BlazeFaceModel | null = null;
                await blazeface.load({ maxFaces: 1 })
                    .then((res) => {
                        console.log("[+] Load Blazeface Model....");
                        _blazefaceModel = res;
                    })
                    .catch((error) => { console.error(`[-] BlazefaceModel Ready Error: ${error}`); });
                setInitModel({
                    isLoading: true,
                    isTensorReady: _tensorReady,
                    blazefaceModel: _blazefaceModel
                })
            },
            readyTensorCamera: async (images: IterableIterator<tf.Tensor3D>, updatePreview: void): Promise<void> => {
                const _loop = async () => {
                    const _imageToTensor: tf.Tensor3D = tf.tidy(() => images.next().value);
                    if (initModel.isTensorReady) {
                        let _prediction: blazeface.NormalizedFace[] = [];
                        if (initModel.blazefaceModel) {
                            // ========================= !!!! Increased graphics memory in that area !!!!! =========================
                            initModel.blazefaceModel.estimateFaces(_imageToTensor, false)
                                .then((res) => {
                                    _prediction = res
                                    res = [];       // 모델을 비움
                                })
                            _prediction = [];
                            initModel.blazefaceModel.dispose();
                        }
                    }
                    _imageToTensor.dispose();       // 사용한 텐서 초기화 
                    timerRef.current = setTimeout(() => _loop(), 600);
                }
                _loop();
            }
        }
    })()
    return (
        <SafeAreaView>
            <StatusBar backgroundColor="#17191c" />
            <ScrollView style={styles.container}>
                <View>
                    {
                        isCameraOn &&
                        <TensorCamera
                            ref={tensorCameraRef}
                            style={styles.camera}
                            type={CameraType.front}
                            ratio={'16:9'}
                            cameraTextureHeight={TEXTURE_DIMS.height}
                            cameraTextureWidth={TEXTURE_DIMS.width}
                            resizeHeight={320}
                            resizeWidth={292}
                            resizeDepth={3}
                            autorender={true} 
                            useCustomShadersToResize={false}
                            onReady={(images, updatePreview) => studyCameraHandler.readyTensorCamera(images, updatePreview())}
                        />
                    }
                </View>
            </ScrollView>
        </SafeAreaView>
    )
}
export default memo(StudyPureScreen);

Please reply. Thank you!

Standalone code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/CodePen/any notebook.

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

[shmishra99]

Hi @adjh54ir ,

I can see in your provided package.json file that you are using older dependencies. Could you please try running it with the latest tfjs@4.22.0? Additionally, blaze-face is deprecated. You could use the face-detection.

Thank You!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No