Using both Audio & Face Detection on video crashed

[alirezanazari]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Android 13

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

No response

Programming Language and version

Kotlin

MediaPipe version

0.10.0

Bazel version

No response

Solution

Face, Audio Speech

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

Android Studio Flamingo | 2022.2.1 - NDK: 23.1.77

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

When i using to record video of user i started to record audio and give it to mediapipe to detect speech accuracy, and after recording finished stop audio detection then start to face detection to get face count. 
at this time `FaceDetector.createFromOptions(context, options)` crashed. when i just comment audio detection codes it worked but both doesn't work sequencely.

Describe the expected behaviour

audio detection and face detection work together.

Standalone code/steps you may have used to try to get what you need

i just used sample codes in this repo.

Other info / Complete Logs

E20230816 13:12:46.917464  6884 graph.cc:472] ValidatedGraphConfig Initialization failed.
No registered object with name: mediapipe::tasks::vision::face_detector::FaceDetectorGraph; Unable to find Calculator "mediapipe.tasks.vision.face_detector.FaceDetectorGraph"
not found: ValidatedGraphConfig Initialization failed.
No registered object with name: mediapipe::tasks::vision::face_detector::FaceDetectorGraph; Unable to find Calculator "mediapipe.tasks.vision.face_detector.FaceDetectorGraph"
com.google.mediapipe.framework.MediaPipeException: not found: ValidatedGraphConfig Initialization failed.
No registered object with name: mediapipe::tasks::vision::face_detector::FaceDetectorGraph; Unable to find Calculator "mediapipe.tasks.vision.face_detector.FaceDetectorGraph"
    at com.google.mediapipe.framework.Graph.nativeStartRunningGraph(Native Method)
    at com.google.mediapipe.framework.Graph.startRunningGraph(Graph.java:336)
    at com.google.mediapipe.tasks.core.TaskRunner.create(TaskRunner.java:72)
        at com.google.mediapipe.tasks.vision.facedetector.FaceDetector.createFromOptions(FaceDetector.java:171)

[alirezanazari]

it's also crash on library latest version

[alirezanazari]

code to reproduce:

package ...

import android.content.Context
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier
import com.google.mediapipe.tasks.audio.core.RunningMode
import com.google.mediapipe.tasks.core.BaseOptions
import com.google.mediapipe.tasks.core.Delegate
import com.google.mediapipe.tasks.vision.facedetector.FaceDetector

class AITest {
    fun init(context: Context) {

        val base = BaseOptions.builder()
        base.setDelegate(Delegate.CPU)

        // Init Face Detector
        val options = FaceDetector.FaceDetectorOptions.builder()
            .setBaseOptions(
                base.
                    setModelAssetPath("face_detection_short_range.tflite")
                    .build()
            )
            .setMinDetectionConfidence(0.7f)
            .setRunningMode(com.google.mediapipe.tasks.vision.core.RunningMode.VIDEO)
            .build()
        val faceDetector: FaceDetector
        try {
            faceDetector = FaceDetector.createFromOptions(context, options)
        } catch (e: IllegalStateException) {
            loge(e)
        } catch (e: RuntimeException) {
            loge(e)
        }

        try {
            AudioClassifier.createFromOptions(
                context,
                AudioClassifier.AudioClassifierOptions.builder()
                    .setScoreThreshold(0.4f)
                    .setMaxResults(2)
                    .setBaseOptions(
                        base
                            .setModelAssetPath("yamnet.tflite")
                            .build()
                    )
                    .setRunningMode(RunningMode.AUDIO_STREAM)
                    .setErrorListener { e ->
                        e.printStackTrace()
                    }
                    .setResultListener { result ->
                    }
                    .build()
            )
        } catch (e: Exception) {
            e.printStackTrace()
        }

    }
}

[kuaashish]

@alirezanazari,

Kindly grant us some time to analyse the potential cause of the issue you are currently encountering. Thank you

[delta42]

I believe I ran across this as well, here, because I too was creating more than one Task. The problem is that some part(s) of MediaPipe are hitting some race condition(s) during initialization.

In your case, the first creation function returns a Promise, and if you wait for it to complete before creating another Task, you might have better luck.

[alirezanazari]

@kuaashish thank you, we need it in our project but for now we disabled audio classification to hear updates from you.

[alirezanazari]

@delta42 no it crash in any cases. in our production codes i initialize and close audio classification 10 seconds before initializing face detection / landmark and unfortunately it crashes.

[alirezanazari]

hi @kuaashish @schmidt-sebastian , do you have any updates about this issue?

[fergushenderson]

No registered object with name: mediapipe::tasks::vision::face_detector::FaceDetectorGraph; Unable to find Calculator "mediapipe.tasks.vision.face_detector.FaceDetectorGraph"

This message generally indicates that your app doesn't have the code for mediapipe.tasks.vision.face_detector.FaceDetectorGraph linked in.

Check your dependencies to make sure that code gets linked in. Do you have a dependency on 'com.google.mediapipe:tasks-vision'? See https://developers.google.com/mediapipe/solutions/vision/face_detector/android#dependencies.

[alirezanazari]

@fergushenderson i told that it works when i comment just one of them(kotlin codes) so dependencies are OK.

[amberguo]

Hi all, I am seeing similar issue, with audio classification and object detection being in the same place. whichever initializes later, it throws out error