Mesh recognition stutters on iOS target

[Ommand]

Visualizer scene looks like this:

Unity 2020.3.24f1 Target: iOS Device: MacBook Pro 2019

The only error in console is

Metal: Error creating pipeline state (Hidden/MediaPipe/FaceMesh/Preprocess): Link failed: The type of fragment input user(TEXCOORD0) does not match the type of the corresponding vertex shader output
(null)

Would be glad to provide any additional information.

[keijiro]

First of all, this project doesn't contain information display that shown on your gif. Are you actually running this project? Are you running a fork/branch in another repository?

[Ommand]

I've added graphy via package manager. This is gif from just cloned repository:

[keijiro]

How about switching to the macOS target?

[Ommand]

Works as intended for PC & Mac Standalone target

[keijiro]

How about changing this:

https://github.com/keijiro/FaceMeshBarracuda/blob/0dce08a144b45a04a2eb195daf8300beffb1a05d/Assets/FaceMesh/Shader/Preprocess.shader#L15-L20

to this?

    float4 Fragment(float4 vertex : SV_Position,
                    float4 uv : TEXCOORD0) : SV_Target
    {
        uv = mul(_Xform, float4(uv.xy, 0, 1)).xy;
        return tex2D(_MainTex, uv.xy);
    }

[Ommand]

It gives the error Shader error in 'Hidden/MediaPipe/FaceMesh/Preprocess': cannot implicitly convert from 'const float2' to 'float4' at line 19 (on metal)

[keijiro]

Okay, then how about this?

    float4 Fragment(float4 vertex : SV_Position,
                    float4 uv : TEXCOORD0) : SV_Target
    {
        float2 uv2 = mul(_Xform, float4(uv.xy, 0, 1)).xy;
        return tex2D(_MainTex, uv2);
    }

[Ommand]

It does not work, gives same error and this warning:

Shader warning in 'Conv2dA_NHWC': signed/unsigned mismatch, unsigned assumed at kernel Conv2DKernel1x1_StrictC16K64_T16x16_R4x4_NHWC at 3/Library/PackageCache/com.unity.barracuda@1.3.3-preview/Barracuda/Runtime/Core/Resources/Barracuda/Conv2d.cginc(773) (on metal)

Also, it has stopped working on other targets as well with this change.

I've checked the case on latest Unity 2021.2.7 and it does not work on Android target too. Maybe Unity bug?

[keijiro]

I have never tested this project on mobile platforms, so there might be compatibility issues on my side.

[yashkhandelwal2006]

@Ommand @keijiro did you find any solution for this yet? I am stuck with this issue as well. Please let me know.

[Ommand]

No, sorry, I didn't. I guess my shader programming understanding isn't good enough.

[wbdsnx]

@keijiro

Hello, are you there

Do you have a better solution to this problem? I have been puzzled by this problem for a long time. I hope to recover as soon as possible. Thank you very much

[keijiro]

@wbdsnx First of all, it's better to use ARKit (AR Foundation on Unity) for face tracking on iOS. Is there any reason to use this package?

I keep this ticked open to track the issue with the package, but I'm not actively working on it because there is a super better alternative (ARKit is the best solution for face tracking on mobile devices).

[wbdsnx]

sorry

First of all, the arkit in arfoundation can only use the front camera, and it can not be well integrated with the mobile capture, so we have to use mediapipe,

We found that Barracuda is relatively lightweight and does not get stuck on the mobile end,

Your technology has always been amazing and reliable

So I sincerely hope you can continue to maintain this repo. Thank you

[wbdsnx]

@Ommand Do you find solution? Please!

[Ommand]

Yes, I've used this package: https://github.com/homuler/MediaPipeUnityPlugin It's a bit tricky to use because you have to compile the plugin for your app, but works pretty good.

[wbdsnx]

@Ommand Thank you !

[mahdipo]

Hi keijiro, Thanks for this repository , It worked on android and windows but on IOS it has error on "Preprocess.shader" at runtime. I fixed it by changing the shader code like below. @Ommand you can replace this code with your self code. please let me know if your problem is fixed too.

Shader "Hidden/MediaPipe/FaceMesh/Preprocess"
{
    Properties
    {
        _MainTex("", 2D) = "" {}
    }

    CGINCLUDE

    #include "UnityCG.cginc"

    sampler2D _MainTex;
    float4x4 _Xform;

    v2f_img vert(appdata_img v)
    {
        v2f_img o;
        o.pos = UnityObjectToClipPos(v.vertex);
        o.uv.xy = v.texcoord;
        return o;
    }

    float4 Fragment(v2f_img i) : SV_Target
    {
        float2 uv2 = mul(_Xform, float4(i.uv, 0, 1));
        return tex2D(_MainTex, uv2);
    }

    ENDCG

    SubShader
    {
        Cull Off ZTest Always ZWrite Off
        Pass
        {
            CGPROGRAM
            #pragma vertex vert // vert_img
            #pragma fragment Fragment
            ENDCG
        }
    }
}

[wbdsnx]

@mahdipo It's well done for me .Thank you very much!

[jsa9292]

Thx for the fixes. Would be nice if the solution is implemented on main. For me this is a better option because it gives customizability. For example, most facial recognition wont return double eye lids, which is big deal in asian faces, but this would allow me to expand the ml portion to have a separate network measuring the eyelids. I can imagine other features built on top of this as well.

[keijiro]

Fixed in the latest update.