search

google-ar / arcore-android-sdk

ARCore SDK for Android Studio
https://developers.google.com/ar
Other
4.98k stars 1.22k forks source link

Augmented Face detection lips sensitivity #1521

Open onuralpszr opened 2 years ago

onuralpszr commented 2 years ago

SPECIFIC ISSUE ENCOUNTERED

Hello, I was developing an app based on kotlin/Arcore related around Augmented Face. While I was doing detection of faces I encountered a problem makes a problem while I was doing occlusion around mouth side of the face. Specifically on upper lips section.

Working example based on mediapipe : https://storage.googleapis.com/tfjs-models/demos/face-landmarks-detection/index.html?model=mediapipe_face_mesh

Also the feature it uses for making right : https://google.github.io/mediapipe/solutions/face_mesh.html#refine_landmarks

When Aug. Face detection done and start tracking, I can clearly see that my upper half of the lips on face mesh goes up. and It also effect that any object I put around mouth basically show more mesh then I didn't wanted to see it.

Arcore detection and drawing 468 nodes

image

Mediapipe detection - Face mesh example without change

image

VERSIONS USED

Packages: versionName=1.32.221520093 Hidden system packages: versionName=0

of packages: 1 / # of packages in DATA: 1 (4)

Active APEX packages: Inactive APEX packages: Factory APEX packages:

Snippet code I use for creating those nodes on Augmented Face 


             private var sphereNodes: ArrayList<Node> = ArrayList()

            for (i in 0..467) {
                val sphereNode = Node()
                val colorSelect = Color(android.graphics.Color.MAGENTA)
                MaterialFactory.makeOpaqueWithColor(
                    context,
                    colorSelect
                )
                    .thenAccept { material: Material? ->
                        sphereNode.parent = this
                        sphereNode.renderable = ShapeFactory.makeSphere(
                            0.0011f, Vector3(
                                0f,
                                0f,
                                0f
                            ), material
                        )
                        sphereNodes.add(sphereNode)
                    }
            }

// For tracking locations ;
// override fun onUpdate(frameTime: FrameTime?) 

            val buffer = augmentedFace!!.meshVertices
            if (buffer != null) {
                sphereNodes.forEachIndexed { index, element ->
                    element.localPosition = Vector3(
                        buffer.get(index * 3),
                        buffer.get(index * 3 + 1),
                        buffer.get(index * 3 + 2)
                    )
                }
            }

I also placed object for testing to see "occlusion" in face, indeed ıt does follow exact same spheres for hiding object or showing object

WORKAROUNDS (IF ANY)

No refine mode

image

With refine mode

image

Source link : https://storage.googleapis.com/tfjs-models/demos/face-landmarks-detection/index.html?model=mediapipe_face_mesh

Maybe there is a way to handle that detection for like "refine-Landmarks" but I didn't find anything yet. Any help would be awesome. Also please let me know If you need anything further

Thank you.

onuralpszr commented 2 years ago

Extra note : I was able to reproduce that problem in "augmented Face example by using" face texture map and I can clearly see that when I smile, "texture moves on top of half my upper lips and think that lower half is "inside of the mouth" as well.

onuralpszr commented 2 years ago

@devbridie any idea ? (friendly ping :) )

kyhoolee commented 1 year ago

I face the same issue too, I think the dev team does not spend much effort for this project. It's really really difficult to combine preview feature and record feature too.

onuralpszr commented 1 year ago

I face the same issue too, I think the dev team does not spend much effort for this project. It's really really difficult to combine preview feature and record feature too.

I found my workaround (not arcore solution) with mediapipe but I customized (face mesh and attention landmark enabled, but issue over there was jitter and I am doing adding filter(like one euro filter, default It was using vector filter for smooting out) and adding extra logic prevent to do that, so It will be act more smoothly but still arcore need something similar with low jitter and lag because this is really problematic

kyhoolee commented 1 year ago