How to render different face effect

[ashrvstv]

In face effect module I can 3d data as glasses.pbtxt facepaint.pngblob glasses.pngblob.

I am trying to add few more models to experiment but i couldnt find any documenation or information of pngblob data. It seems like that using glasses.pbtxt 3d model is generated at runtime and glasses.pngblob is getting used as texture. Can you please clear that is it right and how is it happening

Ques 1- Can you please provide any documentation of pngblob datatype. How can i create new 3d model (pngblob / binarypb ) to render on face. Most common format of 3d model data are OBJ, FBX, etc. Is there any way to convert these format of 3d data to binarypb / pngblob?

Ques 2- it is mentioned in gl_animation_overlay_calculator.cc that .obj.uuu can be created using the mentioned SimpleObjEncryptor but I couldn't find that. can you please specify where to find that ?

ANIMATION_ASSET (String, required):
//     Path of animation file to load and render. Should be generated by
//     //java/com/google/android/apps/motionstills/SimpleObjEncryptor with
//     --compressed_mode=true.  See comments and documentation there for more
//     information on custom .obj.uuu file format.

[kostyaby]

@jclay: Hey! Unfortunately, I don't have an example myself. @duy-maimanh seems to have a working prototype, please reach out to them

@brucechou1983: Hey!

How to balance GPU resource between the two components?

At the end of the day, there is a single GPU device and all you can do is send commands to it. When it comes to balance / bottleneck control, there's significantly less flexibility when compared to CPU. On CPU, you can try creating separate system threads and hope it'll get mapped to separate CPU cores; on some OSes, you can also try to assign priorities to threads and hope to get high-performance CPU cores assigned to threads with more CPU processing work. As far as I know, there's no such thing available for popular GPU APIs

Having said that, you still control how much processing work you send at different stages of the pipeline (perception, rendering), how often and when you synchronize between CPU and GPU. The only way I know to get a well-performing GPU pipeline is to start lean and iterate on bottlenecks in an ad-hoc manner using a CPU/GPU profiler. General advice:

• Extra CPU-GPU syncs are bad (maybe there's a way to remove some of those?)
• Having too much GPU processing work is bad (maybe you can run a lighter perception NN model or render a lighter 3D model with less triangles or pick a lighter shading model with small shader?)

In realtime applications, we use FlowLimiterCalculator in the high level graph to control the number of in-flight streams. If Filament is a separate element of the system, it means we can't have a back edge signal representing the completion of specific timestamp. Is there any possible way to solve this?

If you decide to keep Filament outside of the MediaPipe (MP) graph, then only use FlowLimiterCalculator for what's left inside of the graph: ML perception part. This way, your graph is no longer "frame in - frame out"; instead, it's "frame in - ML signals out". ML signals can a segmentation mask, face geometry proto, hand tracking landmarks, pose tracking landmarks - you name it, all the ML goodness that MP currently provides / can be extended with. Then, your app-level callback is waiting for the geometry, not for the final rendered frame. After that geometry is received, it should be re-packaged as a set of Filament inputs; then you let Filament render your final frame. If you are worried Filament CPU processing would be taking too long synchronously then make it asynchronous (again, separately from the MP graph - the price for having a renderer externally is not being able to use MP synchronization routines)

I wouldn't be crazy about trying to keep everything inside a MP graph. At the end of the day, MP uses the same OS synchronization primitives available to you; even more, it tried to do something very generic (i.e. balance resources in a generic computational graph) while you are trying to do something very specific (i.e. add a single rendering stage on top of the existing perception pipeline). My point is that you are likely to succeed in building a performant "MP perception + external renderer" pipeline :)

Good luck!

[berrymoor]

I was able to make work exactly the approach I proposed earlier: chaining 2 effect renderer one after another. Please, verify that you indeed invoke the correct runtime path by adding logs to Process (not Open) as I recommended in my previous comment.

GIF demo:

Hi, @kostyaby can you share this effect graph file?

[kostyaby]

Hey @berrymoor,

I couldn't find that exact effect graph which I used to generate this GIF, but I've assembled another one in MediaPipe Visualizer that should work as well: combined_effects.txt

Please let me know if it works for you!

[ThilakReddyy]

Hello @kostyaby I'm working on an app that allows me to try on different types of glasses. I use your 2nd suggestion, use separate mediapipe and filament. All working fine. But I have a problem that the parts of the glasses that should be hidden by the face show up over the face (Attached image). Because mediapipe only provide scaling / rotation / translating, I do not know how to calculate the hidden part. Do you have any suggestions?

Hey Could you please share how the rotations are possible with mediapipe

[AswinParanji]

I was able to make work exactly the approach I proposed earlier: chaining 2 effect renderer one after another. Please, verify that you indeed invoke the correct runtime path by adding logs to Process (not Open) as I recommended in my previous comment. GIF demo:

Hi, @kostyaby can you share this effect graph file?

How I can I do this in python?

[kostyaby]

Hey @ThilakReddyy,

Face Geometry module provides support for attaching 3D objects to a face and "inheriting" face 3D rotation. Please, take a look at the documentation and see whether this solution matches your needs

---

Hey @AswinParanji,

Unfortunately, MediaPipe doesn't provide an out-of-box Python API for the Face Geometry module

[ThilakReddyy]

Hello Kanstantsin Sokal, My name is Thilak Reddy and I have a intern at a company and the company wants me to render the 3d objects on face. I have look through the documentation but I am not understanding what to do. Would me please send me a demo project or the required steps to render the image on face.

On Thu, 16 Dec 2021 at 00:19, Kanstantsin Sokal @.***> wrote:

Hey @ThilakReddyy https://github.com/ThilakReddyy,

Face Geometry module https://google.github.io/mediapipe/solutions/face_mesh.html#face-geometry-module provides support for attaching 3D objects to a face and "inheriting" face 3D rotation. Please, take a look at the documentation and see whether this solution matches your needs

Hey @AswinParanji https://github.com/AswinParanji,

Unfortunately, MediaPipe doesn't provide an out-of-box Python API for the Face Geometry module

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1155#issuecomment-995079150, or unsubscribe https://github.com/notifications/unsubscribe-auth/APJGSSOSJVSSSO726F7VIJDURDPM3ANCNFSM4R43BXZQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[duy-maimanh]

Hello @kostyaby I'm working on an app that allows me to try on different types of glasses. I use your 2nd suggestion, use separate mediapipe and filament. All working fine. But I have a problem that the parts of the glasses that should be hidden by the face show up over the face (Attached image). Because mediapipe only provide scaling / rotation / translating, I do not know how to calculate the hidden part. Do you have any suggestions?

Hey Could you please share how the rotations are possible with mediapipe

Mediapipe provide all we need to transform, rotate , scale, etc. This is 4*4 matrix. Sorry but I no longer work with mediapipe so I can't remember exactly. You can read more at here. http://www.opengl-tutorial.org/beginners-tutorials/tutorial-3-matrices/

[kostyaby]

Hey @ThilakReddyy,

We've already shared documentation as well as sample apps for Android and iOS that do exactly what you asked for. I encourage you to make an effort and give it another try. We do not plan to release any additional in-depth materials to cover the topic, maybe someone from the community will do that someday (maybe already have done?)

[JpEncausse]

@kostyaby any sample for the Web in JavaScript ?

[kostyaby]

Hey @JpEncausse,

I didn't have one, so I've forked the MP Face Mesh web example and slapped together a demo using Three.JS: https://codepen.io/kostyaby/full/abqavyo

Not an officially support product, use at your own risk 😉

[JpEncausse]

Perfect ! So cool thanks very much ! I'll give it a try, will let me do AR on many usecases

[Michele-Telesca]

Hey @JpEncausse,

I didn't have one, so I've forked the MP Face Mesh web example and slapped together a demo using Three.JS: https://codepen.io/kostyaby/full/abqavyo

Not an officially support product, use at your own risk 😉

hi i tried to open this example but i get an error. I'm trying to better understand mediapipe and how to integrate it into a web solution with glb model. Could you help me figure out what the problem with the example code is?

[mabab]

Hey @JpEncausse, I didn't have one, so I've forked the MP Face Mesh web example and slapped together a demo using Three.JS: https://codepen.io/kostyaby/full/abqavyo Not an officially support product, use at your own risk wink

hi i tried to open this example but i get an error. I'm trying to better understand mediapipe and how to integrate it into a web solution with glb model. Could you help me figure out what the problem with the example code is?

Some bug with load mediapipe file. You can try use my fix https://codepen.io/Babich/pen/VwXrjvK

[wizVR-zhangjun]

@kostyaby Sorry to bother you, teacher. I want a model of glasses. Can you send it to me? thank you

[wizVR-zhangjun]

@kostyaby I want to know how these glasses are worn on the face, how does it work

[dandingol03]

Hey @JpEncausse, I didn't have one, so I've forked the MP Face Mesh web example and slapped together a demo using Three.JS: https://codepen.io/kostyaby/full/abqavyo Not an officially support product, use at your own risk wink

hi i tried to open this example but i get an error. I'm trying to better understand mediapipe and how to integrate it into a web solution with glb model. Could you help me figure out what the problem with the example code is?

Some bug with load mediapipe file. You can try use my fix https://codepen.io/Babich/pen/VwXrjvK

I can't figure out how does codepen open camera, can you give me some advice~

[dandingol03]

I can't figure out how does codepen open camera, can you give me some advice~

I can't figure out how does codepen open camera, can you give me some advice~

[saad25011]

Hi can any one please guide me how to align glasses. I working on python and using CV2 and media pipe library .Rendering is done but glasses are not completely align with face

[dodorian92]

Hey @JpEncausse, I didn't have one, so I've forked the MP Face Mesh web example and slapped together a demo using Three.JS: https://codepen.io/kostyaby/full/abqavyo Not an officially support product, use at your own risk wink

hi i tried to open this example but i get an error. I'm trying to better understand mediapipe and how to integrate it into a web solution with glb model. Could you help me figure out what the problem with the example code is?

Some bug with load mediapipe file. You can try use my fix https://codepen.io/Babich/pen/VwXrjvK

I'm trying to create a similar example using react. I had no problems doing face reveal via mediapipe. But I think I got stuck in creating EffectRender components for creating the scene and loading the model. Has anyone ever tried to do something like this? or can I find some interesting resource?

[FishWoWater]

@kostyaby @mabab Hi! Thank you very much for your web demo :D I notice that there are some heuristic values in these two lines

gltf.scene.position.set(0, -2, 0);
gltf.scene.scale.set(20, 20, 15);

I understand that they are specified to make rendering of the mask /. wearing close to user faces, but how to nicely determine these values for customized 3D models? Should I first align my 3D model to the canonical face model? (mediapipe/tasks/cc/vision/face_geometry/data/canonical_face_model.fbx)

Thank you very much.

[cgowthamanmca]

@all Please share any example github link please android

[cgowthamanmca]

Hey @jianinz,

I was able to make work exactly the approach I proposed earlier: chaining 2 effect renderer one after another. Please, verify that you indeed invoke the correct runtime path by adding logs to Process (not Open) as I recommended in my previous comment.

GIF demo:

@kostyaby Do you have any example in android app github link if yes please share