Hand tracking is slower than the old Mediapipe for Android

[EmreSafaCelik]

In the old Mediapipe's sample app my test phone could track hands in real time, in this version it is lagging behind. What could be the problem? First thought that comes to my head is analyze case for CameraX being slower. Old Mediapipe sample used an older CameraX version and handled SurfaceView manually if my memory is serving me right. Is there anyway I can get around this and get the same speed for this version? Forgive me if I am asking anything obvious as I am still rather new to Android development.

[EmreSafaCelik]

Perhaps the problem could be this issue here: https://groups.google.com/a/android.com/g/camerax-developers/c/jgeLgaeiO6g

I cannot try on different devices right now. But similar to what is seen on that issue, my device seems to work well with older Mediapipe sample which used an older cameraX, this could be a problem with newer cameraX that affects this sample.

Device I am testing on is Xiaomi Mi 11 Lite

[PaulTR]

Hi @EmreSafaCelik!

So since the Task Library accepts a MPImage, which I'm creating from a Bitmap (I pass in an ImageProxy because of CameraX that I then turn into a Bitmap for preprocessing, but you can change this method around for whatever), I think you'd be OK to try any method that can get an object that's convertible to an MPImage that you want to process asynchronously. I believe MediaPipe Tasks is working the same under the hood while we make it easier to use, but I'll confirm that and update you if I'm wrong there.

That said, I'm honestly not sure if CameraX is slower or not now. Out of curiosity, would you mind sharing what device you're using for testing? I can see if I have something similar laying around, plus I want to reach out to a buddy on the CameraX team to see if they have any insight around it.

Thanks!

[EmreSafaCelik]

Hello Paul, thank you for responding. The device I am mainly testing on is Xiaomi Mi 11 Lite, I have also tested on Xiaomi mi note 9 and 7. I realized the older sample had lower fps in both preview and analyze, but they were in synchrony with each other. I will try using the same thread for both analyze and preview in the new version and see how much of an fps drop I get.

https://user-images.githubusercontent.com/69720413/219100386-7373e480-427a-4db2-a61f-ee866450bda8.mp4

https://user-images.githubusercontent.com/69720413/219100419-cfd9d2b3-de5e-478d-a1e2-f4df82e00a65.mp4

[PaulTR]

Got it, thanks! So I'm able to replicate the slower tracking on my Pixel 7 Pro, too. Inference time is ranging from 50ms to 150ms per frame on my device, so I did reach out to the camera team to see if they're aware of anything under the hood that may have changed. A part of me also wonders if it's my drawing code for the OverlayView being a bit slow, too, but I haven't dug into that.

[EmreSafaCelik]

I tried it this way to quickly test if it would make any difference if draw function was less burden, doesn't feel like it is.

override fun draw(canvas: Canvas) {
        super.draw(canvas)
        results?.let { handLandmarkerResult ->
            val points = mutableListOf<Float>()

            for (landmarks in handLandmarkerResult.landmarks()) {
                for (i in landmarkConnections.indices step 10) {
                    val startX =
                        landmarks[landmarkConnections[i]].x() * imageWidth * scaleFactor
                    val startY =
                        landmarks[landmarkConnections[i]].y() * imageHeight * scaleFactor
                    points.add(startX)
                    points.add(startY)
                }
                canvas.drawPoints(points.toFloatArray(), pointPaint)
            }
        }
    }

https://user-images.githubusercontent.com/69720413/219197860-f276cb00-6434-4e1a-be1f-f48562dc13b8.mp4

Running analyzer on main executor also didn't sync them.

I have been trying to use https://github.com/supremetech/mediapipe-demo-hand-detection to get rid of Bazel for a while and had a similar problem after integrating it to CameraX. It persists here too so I guessed It might be about CameraX. But now I managed to connect some dots and realized you are also using this project, so the problem might be something unrelated as CameraX is not the only thing in common. Though running both preview and analyzer on main executor had somewhat solved this with the price of fps on both in my app. Here it seems not to.

[EmreSafaCelik]

https://issuetracker.google.com/issues/144795164 this is likely the issue I had when I ran both on main executor on my app. The fps dropped a lot when I did that.

[donovanfm]

Hi, @EmreSafaCelik. Are you saying that the FPS drop is lower than the original sample, or is it in line with what the older sample had? If the FPS is too low and it's a problem for you, can you add a comment on the issue you linked to with the devices you tested? That will help us prioritize a fix. Thanks!

[EmreSafaCelik]

Hi Donovan. Though I no longer have problems with the delay for now, It would be great if it was resolved. I added a comment on the issue I linked.

As for whether saying FPS drop is worse than the original, it is not. As the videos above show it, there is a delay in tracking. FPS is fine. Older sample had both the preview and tracking synced as the videos above show, but had worse fps overall.

[PaulTR]

Thanks for the update EmreSafaCelik!

Since it sounds like the issue is on the Android side (with the issue link you posted), I'll leave this thread open for a few more days in case you have other updates before I close it out. Let me know if I'm misunderstanding what's going on though so I don't prematurely cut off a channel :)

Thanks!