How can I utlize multicores present in android to speed up processing

[siddisking]

I have an app which takes a picture from the camera every second and processes it for face detection. I want it to process every image under a second.

I believe we can utilize multicore to speed up the face detection but don't know how.

Any help will be appreciated.

-Thanks

[vcvycy]

It seems that Tensorflow-Android do not support multi-threads. see this But you can try youself by calling function MTCNN::DetectFaces in 2 threads to see if it works. BTW, Tensorflow-Android ( or Tensorflow-Mobile ) is deprecated by official We expect to deprecate TensorFlow Mobile in early 2019 It's better to re-implement the code in Tensorflow-Lite to use more features.

[karloberas]

@vcvycy I've been trying to re-implement your code in Tensorflow-Lite. I followed https://github.com/jiangxiluning/facenet_mtcnn_to_mobile to create pnet.tflite, onet.tflite, and rnet.tflite files and imported to your Android project. I added 'org.tensorflow:tensorflow-lite:0.0.0-nightly' to the gradle dependencies and loaded the models.

I'm stuck at replacing the old code for using inferenceInterface.feed, inferenceInterface.run, and inferenceInterface.fetch to the new method using tfLite.runForMultipleInputsOutputs(inputArray, outputMap). The old code for Tensorflow Android is using float[] while Tensorflow Lite takes in ByteBuffer.

I'm not familiar with using Tensorflow and the documentation is hard to find but I'm very interested to implement your face detection using MTCNN on Android using Tensorflow Lite to improve the performance of our app. This current app using Tensorflow Android keeps crashing because it is using too much system resources.

I would really appreciate your help!

[karloberas]

For additional info, I have implemented your code for real-time camera face detection. That's why any improvement that can be done to the performance is greatly appreciated. I have used Camera2Basic and set minimum face size to 350. Performance is good but the app stopped working from time to time.

[vcvycy]

It seems that we have encounterd the same problem for real-time face detection. But that part is not my work. I will ask how them solve this problem.

[siddisking]

It seems that Tensorflow-Android do not support multi-threads. see this But you can try youself by calling function MTCNN::DetectFaces in 2 threads to see if it works. BTW, Tensorflow-Android ( or Tensorflow-Mobile ) is deprecated by official We expect to deprecate TensorFlow Mobile in early 2019 It's better to re-implement the code in Tensorflow-Lite to use more features.

Just checked it out, its just the thing which I was looking for.... Thanks for the info.

Also I tried reducing the image resolution and it came well under 1000 ms ....

[siddisking]

For additional info, I have implemented your code for real-time camera face detection. That's why any improvement that can be done to the performance is greatly appreciated. I have used Camera2Basic and set minimum face size to 350. Performance is good but the app stopped working from time to time.

Maybe I can help ... What is your requirement?

[karloberas]

@siddisking thanks! I just want to implement the same code using MTCNN for face detection in Tensorflow Lite and I don't know how to proceed.

I'm using this solution for real-time face detection but it seems too heavy for regular devices so my app is getting force closed by the Activity Manager from time to time. I'm thinking using Tensorflow Lite can solve this issue.

[siddisking]

@siddisking thanks! I just want to implement the same code using MTCNN for face detection in Tensorflow Lite and I don't know how to proceed.

You can do real-time detection with this tensorflow logic but it comes with some compromises like image quality and accuracy.

Are you taking pictures with the camera or are you doing a video feed ?

[karloberas]

@siddisking thanks! I just want to implement the same code using MTCNN for face detection in Tensorflow Lite and I don't know how to proceed.

You can do real-time detection with this tensorflow logic but it comes with some compromises like image quality and accuracy.

Are you taking pictures with the camera or are you doing a video feed ?

I'm doing a video feed, sending each frame to the MTCNN detectFace method. Tensorflow Android is also deprecated, that's why I wanted to update it to Tensorflow Lite for better support and improved performance.

[siddisking]

Tensorflow lite seems a way to implement this.

But if tf-lite is not working rn you can achieve 5-15 fps from these hacks:

1. Using multithread to detect faces, you can try this
2. Reduce the size of the TextureView to 352 x 240 so that images are processed faster.

I tried the second fix on an 8-year-old tablet and got 3-5 fps.

Newer Android devices will definitely show some improvement.

I believe with the tf-lite and both these hacks you might be able to achieve over 20 fps (just a guess)

[karloberas]

@siddisking Thank you for the suggestions! I'm currently running MTCNN::detectFaces already in a different thread. Reducing the size of TextureView makes the UI look bad. So it seems I would just have to wait until there is a clear migration guide from Tensorflow Android to Tensorflow Lite or someone can implement MTCNN for Android with TFLite.

[vcvycy]

@karloberas This is how we use camera API to achieve real-time detection. You can try it out.

[karloberas]

@vcvycy thanks for the suggestion but this is simply capturing a photo. The real-time detection that I need is streaming from the camera and processing each frame with MTCNN::detectFaces. I'm already using android-Camera2Basic for this and it is working fine. I just need a way to lighten the processing load on the device when detecting faces so that my app won't be force-closed by Activity Manager.

[kli017]

@karloberas hello. recently I was also consider reimplement the MTCNN in real time face detection. I fellowed the Tensorflow Andriod detection demo and tried to replace the object detector with the MTCNN.. However I was stuck at how to initialize the model. I was wondering could you provide a demo about your implementation? Thank you!

[karloberas]

@kli017 Hi! I believe this repo provides all the code to use the MTCNN model using Tensorflow-Android. You just have to follow the code in the MainActivity.java.

[kli017]

@kli017 Thanks for the reply. I was following the MainActivity.java and trying to modified the DetectionActivity from the TF lite android demo. So If I define a myMain() function as the MainActivity do. The myMain() funciton should do the following right? `public void myMain(){

mtcnndetector=new MTCNN(getAssets());
onPreviewSizeChosen();
processImage();

}`

[sirius0503]

@vcvycy : Can we use mobile gpus in this project to speed up inference?