search

cosyneco / MediaPipe.NET

Pure .NET bindings for Google's MediaPipe.
MIT License
97 stars 18 forks source link

How to extract facial landmarks in xamarin forms app #35

Closed amalcty closed 2 years ago

amalcty commented 2 years ago

Can you provide sample code or documentation to extract facial landmarks from image in a xamarin forms app.

Speykious commented 2 years ago

An example to extract facial landmarks can be found in the MediaPipe.Examples.FaceMesh project in this repository.

As long as you can get the raw bytes of the image in a pixel format that MediaPipe supports (such as RGBA), you will be fine. The output is a simple structure containing 3D landmark coordinates. If you want, you can also use Moetion, our C# port of Kaleidokit, which will give you more well-defined structures such as a Face class, instead of a flat list.

Speykious commented 2 years ago

woops, sorry, didn't mean to do that

amalcty commented 2 years ago

Is there a way to add this in a project targeting netstandard2.1 ?

adryzz commented 2 years ago

no, as we're using .NET 6 features

Speykious commented 2 years ago

It is possible however to make your project target .NET 6 itself. Xamarin should work fine I think.

amalcty commented 2 years ago

No its a whole xamarin forms project based on netstandard 2.1 targeting ios and android, cant change to .net6. MediaPipe.NET.Runtime/Mediapipe.Net.Framework.Protobuf/ targets netstandard2.1 - Can this alone be used for facial landmarks detection ?

Speykious commented 2 years ago

Yes, but you would have to make your own bindings. Also don't forget about the MediaPipe models and graphs which can be found in the original mediapipe repository, they are separate from the source code.

amalcty commented 2 years ago

Okay, thanks. Is there some kind of samples or documentation available for creating such bindings ?

adryzz commented 2 years ago

unfortunately not, otherwise we would have fixed the memory leak much sooner -_-

amalcty commented 2 years ago

Oh okay :(

Speykious commented 2 years ago

Actually there is documentation, it's just that it covers the bare basics, and unless I missed some pages, it doesn't go very deep into ways of solving memory problems.

There's the official Microsoft documentation.

You can also look in the source code of this repository, the pure bindings are in Mediapipe.Net/Native (the code is MIT, if you copy these bindings don't forget to include the license) and every other class that is not in Native uses methods defined there. Though it'll be way more difficult to use MediaPipe this way... It's very easy to create huge memory leaks.

cough totally not talking from past experience cough

amalcty commented 2 years ago

Oops, I don't have the expertise to pull that off.