search

homuler / MediaPipeUnityPlugin

Unity plugin to run MediaPipe
MIT License
1.76k stars 461 forks source link

TongueOut is missing in blendshapes output #1038

Open Hakazaba opened 10 months ago

Hakazaba commented 10 months ago

Description

Hey, im just exploring the amazing new face landmarker blendshape and matrices and i found what seems to be a little bug, the package im working from is a little bit old (#146 on linux) so it might have been fixed, but just encase it isn't i thought it worth mentioning.

The blendshape array lacks 52 - TongueOut. Likely a buffer overflow or something with the array being set to 52 instead of 53

homuler commented 10 months ago

Could you let me know which part of the code you're referring to?

Hakazaba commented 10 months ago

When accessing FaceLandmarkerResult.BlendShapes, it only has 52 indexes, it should have 53.

It may seem like it should be 52 according to this: https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf however 0 is actually 'neutral' and not listed there. You can see this by printing the collection names of each collection in that list

homuler commented 10 months ago

Same as https://github.com/google/mediapipe/issues/4403

By the way, what are you talking about when you mention 'buffer overflow'? Vulnerability?

Hakazaba commented 10 months ago

Never mind, i had thought the model would be outputting 53 blendshapes, then inserted into an array with a length of 52, thereby causing a buffer overflow and a loss of the final blendshape, due to the documentation saying there are only 52 blendshapes without including the 'neutral' blendshape at 0. It seems the problem is more endemic than that.