Blazeface result is not correct on WASM backend

[deepanshusharma012]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): Yes, I am using the estimateFace function of blazeface to detect the actual face in the image uploaded. Here is my code for reference:

  const detectFace = async () => {
    const p = document.getElementById('result');

    p.innerHTML = "Detecting ...";
    console.log('Loading model...')
    const model = await blazeface.load({scoreThreshold: 0.9});
    console.log('Loaded model...')

    var predictions;
    var t0, t1;

    for (j = 0; j < 5; j++) {
        var t0 = performance.now()
        // Pass in a video stream (or an image, canvas, or 3D tensor) to obtain an
        // array of detected faces from the MediaPipe graph. If passing in a video
        // stream, a single prediction per frame will be returned.
        predictions = await model.estimateFaces(canvas, false);
        console.log(predictions);
        var t1 = performance.now()
    }
    if (predictions.length > 0) {
        var prediction = predictions[0]
        var text = "The face detection completed in " + (t1 - t0) + " ms.<br>The face matched with predictions of " + predictions[0].probability[0];
        text += "<br>"
        text += `<strong>Right Eye:</strong> <br> x = ${prediction.landmarks[0][0]}; y = ${prediction.landmarks[0][1]}<br>`
        text += `<strong>Left Eye:</strong> <br> x = ${prediction.landmarks[1][0]}; y = ${prediction.landmarks[1][1]}<br>`
        text += `<strong>Nose:</strong> <br> x = ${prediction.landmarks[2][0]}; y = ${prediction.landmarks[2][1]}<br>`
        text += `<strong>Mouth:</strong> <br> x = ${prediction.landmarks[3][0]}; y = ${prediction.landmarks[3][1]}<br>`
        text += `<strong>Right Ear:</strong> <br> x = ${prediction.landmarks[4][0]}; y = ${prediction.landmarks[4][1]}<br>`
        text += `<strong>Left Ear:</strong> <br> x = ${prediction.landmarks[5][0]}; y = ${prediction.landmarks[5][1]}<br>`
        renderPrediction(predictions);
    } else {
        var text = "No face found in the image.";
    }
    p.innerHTML = text;
}

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): This behaviour is reproducible across multiple OS of both desktop & mobile devices.

• TensorFlow.js installed from (npm or script link): https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@3.1.0

• TensorFlow.js version (use command below): 3.1.0

• Browser version: this behavior is reproducible across multiple Browser versions including latest Chrome 92

Describe the current behavior I am using the estimateFace function of blazeface to detect the actual face in the image uploaded. It is giving 0.9804430603981018 probability when I placed the mobile device image to detect. If the face is present in the frame, then it detects properly the face and results are as expected. But if we don't have any face in the frame, in that case the blazeface detection results shows that face exists while it is Mobile device with 0.9804430603981018 probability with 0.9 scoreThreshold. (Attaching the image for reference)

Describe the expected behavior It should show result of face detected only if any face is present in frame. If there is no face in frame, then there is no point to show face detected with such high probability i.e. of 98% for mobile device on frame.

Standalone code to reproduce the issue You can use the following codepen link to reproduce: https://codepen.io/deepanshusharma012/pen/MWmRaLW Attaching the mobile device pic on which the issue is reproducible, Also the blazeface models that I'm using for face detection.

Issue Reproducible Snapshot:

Image to reproduce the issue

Blazeface models used in face detection: blazeface models.zip

[deepanshusharma012]

@lina128 @rthadur Hope you are doing great! Is there any update on this issue or any ETA to resolve it? Actually this issue is a blocker for our application and detecting the false events. It would be great if you respond on it and try to roll it's fix asap.

[lina128]

Hi @deepanshusharma012 , thank you for reporting this. I tested it with our own demo, same problem. Actually we're working on a new BlazeFace model, which is going to be more accurate. It will be released this quarter.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[deepanshusharma012]

The work is still going on this issue. I hope we'll be getting activity on this issue, so no need to mark it stalled.

[deepanshusharma012]

Hi @lina128, as you said the new BlazeFace model (more accurate one) will be released by Q3 quarter. So, are we still working on it, or it has released? or any ETA for the same.

[lina128]

Sorry, I actually meant Q4. I replied in September, and we are already thinking for Q4 :) Sorry for the confusion.

[deepanshusharma012]

@lina128 Is there any update on the new BlazeFace model?

[lina128]

Added @ahmedsabie who is working on it.

[deepanshusharma012]

@lina128 Thanks for the update.

@ahmedsabie Do we have any ETA to get the new BlazeFace model (more accurate one) released?

[ahmedsabie]

hey @deepanshusharma012 we are aiming for sometime this month

[rthadur]

@deepanshusharma012 did you get chance to check latest blazeface model here https://github.com/tensorflow/tfjs-models/blob/master/face-detection , it is now deployed.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[ayadav]

@rthadur will test shortly and share results.

[rthadur]

Closing this issue , please @mention to reopen, if the issue still persists or open a new issue with latest information. Thank you

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[rahulgarg03]

Hi @rthadur , the above issue still persists for latest blazeface models having modelType set as 'short' (2 metres). Will there be any updates for the same? It is currently working fine for model type set as 'full'(5 metres). Image for which the issue persists:

[lina128]

We have deprecated blazeface API, please use face-detection instead: https://github.com/tensorflow/tfjs-models/tree/master/face-detection

[rahulgarg03]

Hi @lina128, I have used the face-detection model. But in this face-detection model for modelType set as 'short' (2 metres) the issue still persists. Will there be any updates for the same?

[lina128]

Thank you for testing out the new API, I will investigate.

[rahulgarg03]

Hi @lina128 Is there any update on the above issue?

[gaikwadrahul8]

Hi, @deepanshusharma012

Apologize for the delayed response and we're re-visiting our older issues and checking whether those issues got resolved or not as of now so May I know are you still looking for the solution or your issue got resolved ?

If issue still persists after trying with latest version of TFJs and please refer this official documentation please let us know with error log and code snippet to replicate the same issue from our end ?

Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ? Thank you!

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[SarthakBhatia2018]

Hi @gaikwadrahul8 , For our current use case we are majory targetting using this for webcam and front facing cameras on smartphone, so as per the documentation model type should be set as short. [https://drive.google.com/file/d/1d4-xJP9PVzOvMBDgIjz6NhvpnlG9_i0S/preview]

I have tested the same with the latest face detection models for above image. It is giving the same issues when tested for modelType as short. In case we set the model type as full it works correctly.

[gaikwadrahul8]

Hi, @SarthakBhatia2018

Apologize for the delayed response and I hope you're aware of tfjs-models/blazeface is deprecated in favor of the new face-detection package as per the official documentation so please try to use new face-detection package if issue still persists with new face-detection then please let us know with error log or with output of image. Thank you!

[SarthakBhatia2018]

Hi @gaikwadrahul8,

Checked the same with mediapipe-gpu, tfjs-webgl and tfjs-wasm backends. Attaching the results with images :

with mediapipe-gpu

with tfjs-wasm :-

with tfjs-webgl

Let me know if any addtional information is required.

[SarthakBhatia2018]

Hi @gaikwadrahul8 let me know if there is any update on above.

[gaikwadrahul8]

Hi, @SarthakBhatia2018

Apologize for the delayed response and could you please help us with your codepen example which you tried with latest version of @tensorflow/tfjs@4.8.0 or previous stable version of it and @tensorflow-models/face-detection so I'll try to replicate the same issue from my end ? Thank you!

[SarthakBhatia2018]

Hi @gaikwadrahul8, I have added the code for testing face detection by uploading images. You can clone the following repository repo-link and switch to upload_image brach. Navigate to face-detection -> demos -> upload_image. folder-link

The steps to run it are the same as other folders in demos. Refer demo

Let me know if anything else is required from my end.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[SarthakBhatia2018]

Hi @gaikwadrahul8 let me know if there is any update on the same.

[gaikwadrahul8]

Hi, @SarthakBhatia2018

Apologize for the delayed response and I tried one random image from the internet with person face hiding behind the smartphone and in my case tfjs-webgpu runtime backend is giving me face detection as result so it seems like there some unexpected behaviour happening at the moment so we'll dig more into this issue and will update you soon , For your reference I have added screenshot below with modelType :short

Thank you for noticing this issue, I really appreciate your efforts and time. Thank you!

Runtime backend : mediapipe-gpu

Runtime backend : tfjs-webgl

Runtime backend : tfjs-wasm

Runtime backend : tfjs-webgpu

[SarthakBhatia2018]

Hi @gaikwadrahul8, let me know if the above issue is fixed.