How to render different face effect

[ashrvstv]

In face effect module I can 3d data as glasses.pbtxt facepaint.pngblob glasses.pngblob.

I am trying to add few more models to experiment but i couldnt find any documenation or information of pngblob data. It seems like that using glasses.pbtxt 3d model is generated at runtime and glasses.pngblob is getting used as texture. Can you please clear that is it right and how is it happening

Ques 1- Can you please provide any documentation of pngblob datatype. How can i create new 3d model (pngblob / binarypb ) to render on face. Most common format of 3d model data are OBJ, FBX, etc. Is there any way to convert these format of 3d data to binarypb / pngblob?

Ques 2- it is mentioned in gl_animation_overlay_calculator.cc that .obj.uuu can be created using the mentioned SimpleObjEncryptor but I couldn't find that. can you please specify where to find that ?

ANIMATION_ASSET (String, required):
//     Path of animation file to load and render. Should be generated by
//     //java/com/google/android/apps/motionstills/SimpleObjEncryptor with
//     --compressed_mode=true.  See comments and documentation there for more
//     information on custom .obj.uuu file format.

[kostyaby]

Hey @ashrvstv,

Thanks for your interest!

PNGblob

In fact, it is a regular PNG format; .pngblob is used instead of .png to prevent iOS build from preprocessing the image. OpenCV is unable to read a PNG file preprocessed by the iOS build. So you can use any filename.png for your effect, just rename it filename.pngblob

3D Model

Ultimately, the model .pbbinary file must follow the mediapipe.face_geometry.Mesh3d binary protobuf format in order to be rendered. Here's a path for turning an OBJ 3D model into that format step-by-step:

• Convert an OBJ model into mediapipe.face_geometry.Mesh3d text protobuf format. We do not provide an official OBJ -> Mesh3d pbtxt converter, but I can share a non-official one that I wrote to convert the glasses OBJ asset: link
• Use encode_binary_proto Bazel rule to convert model.pbtxt into model.pbbinary: link

If you have your 3D models in a format other than OBJ, then please try to convert those into OBJ first. Please note, that your model must not have more than 65535 (2 ^ 16 - 1) vertices due to the effect renderer limitations.

The gl_animation_overlay_calculator.cc question

I believe that converter is not available in this repository, sorry about that! You can understand what format is expected by reading its parsing logic: link. Then, you might implement your own converter. I believe, there's nothing we can easily share with you right now to ease your workload :/

[at-duymai]

Hello. @kostyaby I have a question. Usually 3d softwares often export OBJ file with MTL file. How can I create a PNG file of glasses like in the demo Faceeffect?

[kostyaby]

Hey @at-duymai,

One of the reasons I decided against supporting OBJ format directly was exactly the fact that we'd have to support MTL material shading and it felt too complex for a basic example. Therefore, we do support OBJ topology format via converting into mediapipe.face_geometry.Mesh3d via the non-official converter; however, we only support the simplest color texture shading model, so there's no need for MTL file as all those diffuse / reflectivity / other parameters don't make sense for the color texture shading model.

To create an asset for our face effect renderer, you first need to model it in some 3D asset authoring tool (Blender / Maya / Cinema 4D). After the asset is modeled, you need to export its topology as OBJ and then convert it into mediapipe.face_geometry.Mesh3d using the tool; as for the asset material part, you just need to export its color texture (remember, our shading model only support that component) as PNG and hook it up into the renderer

[breathim]

@kostyaby how to load the images like facepaint.pngblob dynamically from a file storage(google drive) ? or is there any provision to change the color of the texture dynamically which is marked in the .pngblob file.

[kostyaby]

Hey @breathim,

I believe that dynamic asset management is an orthogonal functionality which is well beyond our face effect example, as it lies in the realm of general mobile/desktop/web platform app development. At this point, MediaPipe doesn't provide any built-in solution for that and is instead strictly about running image / video processing pipelines

As long as you are able to download an asset and pass the correct path to the renderer, it'll have no problem reading / interpreting those instead of app-bundled assets.

[ashrvstv]

@kostyaby what do I need to modify in case I want to add more complex 3D models with lighting (e.g. diamond). In that case color texture won't be enough. So how can I add support for mtl files?

[kostyaby]

Hey @ashrvstv,

This is an example renderer, that showcases some basic techniques of face AR. At the current moment, we do not intend to make it as comprehensive as what you might expect from other production-quality AR renderers like Snapchat / Facebook has in their user-facing products

If you are interested in mobile platforms, I'd recommend you to take a lot at https://github.com/google/filament. It shouldn't be hard to drive the Filament renderer using mediapipe.face_geometry.FaceGeometry to create effects of a much better visual quality.

[WaelShaikh]

Hey @kostyaby, is there any example or documentation for using the effect renderer for face effects with the tensorflowJS model?

[k-konovalov]

@ashrvstv You must create your own file-calculator which implement filament as a renderer. I work at this, but it's a little bit complicated)

[kostyaby]

@GamerWael, there is none; I'm not even sure MediaPipe has a TensorflowJS inference calculator as TensorflowJS itself is not a native library

You could theoretically make a calculator like that only for Web platform and use Emscripten bindings to pass data between native WASM binary and TensorflowJS. At this point, I'm not sure it worth it as Tensorflow Lite inference calculator is being beautifully compiled into WASM and has Web-specific accelerations via XNNPACK

@konovalov-k, I've been looking into that a few months ago; if you are interested, I'll share my progress notes here:

• There are 2 ways Filament can be integrated with MediaPipe: 1) as a calculator and 2) as a separate mobile app component, that passes signals from MediaPipe graph into the Filament renderer
• The challenging part about completing 1) is that MediaPipe and Filament have pretty different ways to get GPU initialized; Filament needs a native surface, MediaPipe doesn't provide that explicitly.
• In the end, I was able to only make the OpenGL backend work by providing my own backend::OpenGLPlatform implementation, which is essentially no-op and thus allows Mediapipe to handle GPU initialization (PlatformWebGL is what I used as a basis for my custom platform). Other backends (Metal on iOS, Vulkan on modern Android) wasn't possible to hook up with the way Filament is currently implemented; I chatted with the framework team a bit and generally they were not super happy that I was meddling with custom GPU initialization and warned me that there might be potential future issues as they do not expect to provide support at such low-level of the API.
• TLDR on 1): it is possible to integrate Filament as calculator, but it's challenging, fragile and only gives you access to the old OpenGL ES 3.0 graphics backend.
• This brings me to the 2) solution. You set up MediaPipe graph & Filament renderer separately in your app; then, set up a mechanism for passing face tracking results from the MP graph into the renderer (if you have apps for Android + iOS, then you'd have to write this logic twice OR extract common pieces into a native C++ library of your own). I haven't tried it myself, but it sounds way easier as it mostly about plumbing code that doesn't meddle with GPU

Hopefully this was helpful!

[jianinz]

Hi @kostyaby, i followed your suggestions with steps to convert another 3D glass OBJ into mediapipe.face_geometry.Mesh3d text protobuf format, but somehow when load it in Android, the glass was partially seen (half of the glass) and it does not attach to the the eye area, that makes me wonder maybe i am missing some critical configuration to define the boundary or using landmarks to define where the glass should go to?

[kostyaby]

Hey jianinz@,

It sounds to me like your 3D glasses model wasn't aligned with the canonical face mesh 3D model (mentioned here under the Bridges static and runtime spaces point; model OBJ)

Can you please confirm, that when both the new glasses model & the canonical face model are imported into the same 3D scene, the glasses are "aligned" with the face? You can check that by, for example, importing both models into some 3D software tool (Blender, Maya etc)

[jianinz]

Hi @kostyaby, you are correct, they are not aligned, the new glass model is way too small compare to the canonical face 3D model. Thanks!

[jianinz]

Hi @kostyaby , given the example of face effect is to use GateCalculator to decide whether it's face paint or glass effect to be rendered. I wonder that is it possible to mux different face effects (glass, earrings, necklace etc) at the same time? I have tried to use multiple FaceGeometryEffectRendererCalculator within each having its own effect texture and mesh_3d, in the end using ImmediateMuxCalculator to mux them, unfortunately it does not work as expected. Do you know if i missed something important here? Thanks

[kostyaby]

Hey @jianinz,

Currently in the example, there are 2 effects guarded by a single is_facepaint_effect_selected flag. If it's true, then the Facepaint effect will be rendered; if it's false, then the Glasses effect will be rendered. When you switch from 2 effects to 3+ effects, this single boolean flag approach no longer works

For N effects, I'd introduce N graph input streams with boolean flags. Ex: is_facepaint_effect_selected, is_glasses_effect_selected, is_necklace_effect_selected etc. With every input frame, you must then sent all these N boolean flags into the graph and exactly one of them must be true

Alternatively, you can implement a new calculator in C++ that picks the effect based on the index - this way, you'll be able to keep the number of additional graph input streams to zero by essentially replacing the is_facepaint_effect_selected boolean flag packet with selected_effect_idx integer packet.

[jianinz]

Thanks @kostyaby for your suggestions. Will look into that!

[Nvelu048]

Hi @kostyaby i followed your below reply and generated model.pbtxt

basically i used this by converting glb to obj

Hey @at-duymai,

One of the reasons I decided against supporting OBJ format directly was exactly the fact that we'd have to support MTL material shading and it felt too complex for a basic example. Therefore, we do support OBJ topology format via converting into mediapipe.face_geometry.Mesh3d via the non-official converter; however, we only support the simplest color texture shading model, so there's no need for MTL file as all those diffuse / reflectivity / other parameters don't make sense for the color texture shading model.

To create an asset for our face effect renderer, you first need to model it in some 3D asset authoring tool (Blender / Maya / Cinema 4D). After the asset is modeled, you need to export its topology as OBJ and then convert it into mediapipe.face_geometry.Mesh3d using the tool; as for the asset material part, you just need to export its color texture (remember, our shading model only support that component) as PNG and hook it up into the renderer

Now, how to export color texture as PNG.

Or where to find color texture for 3d model

Need your guidance

[kostyaby]

Hi @Nvelu048,

Given that Glb is a format provided alongside with glTF, I'd be looking for a corresponding texture converter somewhere around tooling provided by Khronos.

For that Deep Cowboy Medical Face Mask model in particular, you can alternatively choose to download it as Original / Updated glTF from the Poly website and use the provided JPG texture as-is (our renderer should supports both PNG & JPG)

[Nvelu048]

Hi @kostyaby

as per your guidance, i did as follows:

1. Downloaded Deep Cowboy Medical Face Mask and used it's jpg texture
2. For pbtxt file, i have imported glTF file in Blender and aligned it in accordance with canonical_face_mesh and exported it as obj file
3. Then, used the provided script to generate pbtxt file.
4. Then updated the graph and started the app. In log i could find assets were successfully loaded.

But still effect doesn't gets rendered

I don't know where i am going wrong

Drive Link having resources i used

[Ravipuniya]

HI @kostyaby i want to export the color texture in png format using blender but enable to figure out how to get from blender. Could anyone please help me. I am able to get .mtl file from blender while exporting .obj from blender.

[kostyaby]

Hey @Ravipuniya,

Have you tried googling? Something along these lines seems relevant: link.

There's of course a possibility, that there are no underlying textures for a given material, but it's rather represented through parameters. In this case, you are looking for a way to "bake" that material into a texture. Something like this seems relevant for this case: link 1, link 2

---

Hey @Nvelu048,

I think the alignment step might be incorrect. I imported both the mask_aligned.obj and canonical_face_mesh.obj into Blender (v 2.79) and they are not aligned. Please check out the following screenshots:

Can you please confirm that importing both *.obj models into your Blender environment produces the same result? If so, then you might want to re-align the 3D model and then re-export.

[brucechou1983]

I'm also trying to figure out how to add new face paint effects to the example app.

It seems to that facepaint.pngblob is aligned with canonical_face_model_uv_visualization.png. For new face paint designs, do we need to manually perform this process? I guess the flow is like:

1. manually label some key points like eyes, mouth, nose and edge of the mask
2. scale and warp the mask to align with the canonical face uv map

Do I miss anything or is there better way to do this? Thanks.

[Nvelu048]

Hi @kostyaby

When i imported mask_aligned.obj file in blender-2.91.0

It was aligned as below in layout mode

I used canonical_face_mesh.fbx, which i got from ARCore Project.

Am i referring the wrong canonical_face_mesh. If so, Kindly share the obj format

[kostyaby]

Hey @brucechou1983,

You are right that the face paint texture should be aligned with UV coords of the canonical face mesh (visualization of which is available in canonical_face_model_uv_visualization.png).

The procedure for generating new face paint texture described by you looks fine to me. Maybe it will simplify the first step if I share this psd file from AR Core repo. AR Core runs a similar backbone for its Augmented Faces API as MediaPipe Face Geometry API; that texture in particular is perfectly aligned with our UVs

---

Hey @Nvelu048,

You are right that AR Core canonical face model is likely positioned differently. According to this, they 1) use 0.01 scale along all axis - which explains why on my screenshots the face mask is so tiny - and 2) use left-handed 3D space (the one with +Z axis instead of -Z axis) - which explains why the face mask is rotated along Y axis.

Besides that psd file that I shared in my response to @brucechou1983, please avoid mixing and matching assets between MediaPipe Face Geometry and AR Core Augmented Faces libraries as they do not necessarily match; You can find assets for the Face Geometry API here

[Nvelu048]

Hi @kostyaby Thanks a lot. Able to render 3d effects as expected.

[afsaredrisy]

@kostyaby Can you please share some details how you correctly align mask to canonical face model. Or Can you share your .obj and .pngblob file so that I can check it in Blender. Will be thankful

[jianinz]

Hi @kostyaby It's possible to render another 3d effect, and i wonder if it's possible to render multiple 3d effects at the same time with each of them having different color texture? for instance, i'd like to have a pair of glass together with a horn (i know it's odd combination :)) with both of them having different color texture

[kostyaby]

Hi @afsaredrisy,

I'm not an expert in 3d modeling, but the goal is to make sure the 3d asset you have "fits" the canonical face 3d model; This can achieved by scaling / rotation / translating the 3d asset until it meets the goal criteria. If you are looking for a 3d modeling tool, then I'd recommend starting with Blender as it's free, has reasonable docs and can get you there

There's an example OBJ asset available here; if I remember correctly, it's the red glasses asset that you see in the example app, so you can use this texture. Alternatively, many 3d editors allow you to work with a 3d model without a texture, so you might not need a texture at all

---

Hi @jianinz,

The way it's currently implemented, your only option is to stack multiple FaceGeometryEffectRendererCalculators one after another to first render a pair of glasses and then a horn. However, if you could "merge" color textures, UV coordinate space and XYZ positions for those two 3d models into one model, then you could get by a single FaceGeometryEffectRendererCalculator.

I'm sorry about the inconvenience! This renderer was built as an simple example, so it support the bare minimum of functionality :/

[jianinz]

Hi @kostyaby thanks for your suggestion! I have tried to stack multiple FaceGeometryEffectRendererCalculators, although ImmediateMuxCalculator seems do not mux those effects at all, it seems just select one or another effect like in the face effect example app. Am i missing something?

[kostyaby]

Hi @jianinz,

What you need to do is to re-assign some input / output stream. Let's take face_effect_gpu.pbtxt as an example: if you want to first render the Facepaint effect and then the Glasses effect, then then output stream of the first renderer calculator (i.e. output_stream: "IMAGE_GPU:facepaint_effect_output_video") must go as the input stream of the second renderer calculator (i.e. input_stream: "IMAGE_GPU:facepaint_effect_output_video").

Please note, that you'll likely need multiple instances of the same FaceGeometryEffectRendererCalculator doing the same effect (like Glasses) for each of the possible paths (like, only Glasses and Facepaint + Glasses). The reason for that is that it's not possible to dynamically re-assign input / output stream names in MediaPipe, so the renderer calculator for Facepaint effect can only output into either facepaint_effect_output_video or glasses_effect_throttled_input_video

Let me know if you have questions!

[jianinz]

Please note, that you'll likely need multiple instances of the same FaceGeometryEffectRendererCalculator doing the same effect (like Glasses) for each of the possible paths (like, only Glasses and Facepaint + Glasses).

Thanks @kostyaby Do you mean the actual 3D model needs to be Facepaint + Glasses?

What i did was:

# Renders the eyepatch effect.
node {
  calculator: "FaceGeometryEffectRendererCalculator"
  input_side_packet: "ENVIRONMENT:environment"
  input_stream: "IMAGE_GPU:throttled_transformed_input_video_gpu"
  input_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
  output_stream: "IMAGE_GPU:eyepatch_effect_output_video"
  node_options: {
    [type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
      effect_texture_path: "mediapipe/graphs/face_custom/data/eyepatch.pngblob"
      effect_mesh_3d_path: "mediapipe/graphs/face_custom/data/eyepatch.binarypb"
    }
  }
}

# Renders the pirate effect.
node {
  calculator: "FaceGeometryEffectRendererCalculator"
  input_side_packet: "ENVIRONMENT:environment"
  input_stream: "IMAGE_GPU:eyepatch_effect_output_video"
  input_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
  output_stream: "IMAGE_GPU:output_video"
  node_options: {
    [type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
      effect_texture_path: "mediapipe/graphs/face_custom/data/pirate.pngblob"
      effect_mesh_3d_path: "mediapipe/graphs/face_custom/data/pirate.binarypb"
    }
  }
}

as you can see that the input stream of second FaceGeometryEffectRendererCalculator takes in the output of first FaceGeometryEffectRendererCalculator. But this gives me only the second renderer effect.

[kostyaby]

Hey @jianinz,

Yeah, you did exactly what I purposed by chaining 2 renderer calculators. It's unexpected that you only see the second effect :/

Is there a standalone pirate effect in the graph? If so, can you please confirm that this eyepatch + pirate chained effect path is triggered instead of the just pirate effect path? Also, can you please confirm that the eyepatch effect can be renderer stand by itself?

[jianinz]

Hi @kostyaby ,

yeah, it's odd, i checked it again, there is a standalone pirate effect in the graph, adding log in effect_renderer_calculator.cc shows that both effects were read from correct path. If i just have standalone eyepatch effect, it can be correctly rendered.

The complete graph regarding the render effect nodes are:

# Renders the standalone eyepatch effect.
node {
  calculator: "FaceGeometryEffectRendererCalculator"
  input_side_packet: "ENVIRONMENT:environment"
  input_stream: "IMAGE_GPU:throttled_transformed_input_video_gpu"
  input_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
  output_stream: "IMAGE_GPU:eyepatch_effect_output_video"
  node_options: {
    [type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
      effect_texture_path: "mediapipe/graphs/face_custom/data/eyepatch.pngblob"
      effect_mesh_3d_path: "mediapipe/graphs/face_custom/data/eyepatch.binarypb"
    }
  }
}

# Renders the standalone eyepatch effect.
node {
  calculator: "FaceGeometryEffectRendererCalculator"
  input_side_packet: "ENVIRONMENT:environment"
  input_stream: "IMAGE_GPU:throttled_transformed_input_video_gpu"
  input_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
  output_stream: "IMAGE_GPU:pirate_effect_output_video"
  node_options: {
    [type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
      effect_texture_path: "mediapipe/graphs/face_custom/data/pirate.pngblob"
      effect_mesh_3d_path: "mediapipe/graphs/face_custom/data/pirate.binarypb"
    }
  }
}

# Renders all together effect.
node {
  calculator: "FaceGeometryEffectRendererCalculator"
  input_side_packet: "ENVIRONMENT:environment"
  input_stream: "IMAGE_GPU:eyepatch_effect_output_video"
  input_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
  output_stream: "IMAGE_GPU:all_together_output_video"
  node_options: {
    [type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
      effect_texture_path: "mediapipe/graphs/face_custom/data/pirate.pngblob"
      effect_mesh_3d_path: "mediapipe/graphs/face_custom/data/pirate.binarypb"
    }
  }
}

node {
  calculator: "ImmediateMuxCalculator"
  input_stream: "all_together_output_video"
  input_stream: "pirate_effect_output_video"
  output_stream: "output_video"
}

Please correct me if anything wrong with this, thanks!

[kostyaby]

Hey @jianinz,

there is a standalone pirate effect in the graph, adding log in effect_renderer_calculator.cc shows that both effects were read from correct path

Having correct assets is not enough to confirm that the correct path is taken in runtime. You need to add logging into Process and make sure, that when you expect to see the "eyepatch + pirate" effect you see both calculators printing into logs

If that doesn't work, I'll take a look during week and get back to you

[kostyaby]

Hey @jianinz,

I was able to make work exactly the approach I proposed earlier: chaining 2 effect renderer one after another. Please, verify that you indeed invoke the correct runtime path by adding logs to Process (not Open) as I recommended in my previous comment.

GIF demo:

[jianinz]

Thanks for checking @kostyaby ! Appreciated it. I will double check on my side and get back to you. Thanks!

[bugmany]

i want to how to create OBJ file and the relation of PNG file. thanks

[kostyaby]

Hey @bugmany,

If I understood your question correctly, it has been discussed in great length in comments to this issue so I encourage you to take a closer look there first. If that's not helpful, please be more specific which parts of the ongoing discussion require further clarification

[bugmany]

thanks @kostyaby, i can create those files.

[duy-maimanh]

Hello @kostyaby I'm working on an app that allows me to try on different types of glasses. I use your 2nd suggestion, use separate mediapipe and filament. All working fine. But I have a problem that the parts of the glasses that should be hidden by the face show up over the face (Attached image). Because mediapipe only provide scaling / rotation / translating, I do not know how to calculate the hidden part. Do you have any suggestions?

[kostyaby]

Hey @duy-maimanh,

Good job on making a MediaPipe + Filament prototype! The final thing that you need is a face occluder in the scene. Please, grab the MediaPipe canonical face 3D model (FBX, OBJ), add it into your Filament scene and render it as an occluder (read/write in depth buffer, don't write anything into color buffer). Not sure how exactly it could be done with Filament, but what helped me with other 3D engines is enforcing the Opaque mode for the occluder entity + setting alpha = 0. I'm pretty sure occluders are possible with Filament

[NourOmran]

Greetings to all of you , I want to make an AR app to try sunglasses so I used mediapipe python to get the landmarks , I have glasses.obj but I do not know what is the next step to make the 3d projection , I used openCV but it doesnot work and I tried a lot of stuff, what should I do ? and will it be easier if I will use ArCore ? and which is better client side or server side ? thanks

[kostyaby]

Hey @NourOmran,

MediaPipe still lacks Face Geometry support for Python, so the guidance in gave in this comment is still valid; please take a look to see whether it's helpful to you

Regarding ARCore, I don't think they give you any Python at all, so I don't think it'll match the Python requirement.

Regarding client vs server, it depends on many factors, but normally people prefer doing AR on client due to latency & server cost concerns.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[midhulavijayan]

Hi @kostyaby,

I am developing an app to try different types of glasses. As per your guidance, i tried to change the asset files. For that i have downloaded obj files from "https://www.cgtrader.com/3d-models/eyeglasses" and converted using the python script which you have mentioned. But the glass is not fitting on the face properly. Could you please guide me to do this?.. I am very new to 3d modelling. Can you please share different asset files for glass effect?

[kostyaby]

Hey @midhulavijayan,

One important step for making sure a 3D glasses asset is aligned with face at runtime is to align the 3D glasses asset with the canonical face model (similar comments: https://github.com/google/mediapipe/issues/1155#issuecomment-739078145, https://github.com/google/mediapipe/issues/1155#issuecomment-744628191, https://github.com/google/mediapipe/issues/1155#issuecomment-767008953). In addition to my recommendation from the last comment, I'd recommend using https://threejs.org/editor/: just import both the canonical model OBJ file and your glasses model into the workspace, align the glasses model with the canonical model by moving / rotating / scaling and then export the resulting glasses OBJ model

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[midhulavijayan]

Hi @kostyaby, Thanks. Finally it worked😀

[jclay]

@kostyaby Any chance you have some code in the wild that demonstrates integrating MediaPipe with a 3D engine such as Filament?

And to anyone in the open source community who is interested – we have a budget to fund work on an open source demo of a MediaPipe + external 3D engine integration. You can find my email in my profile.

[brucechou1983]

@kostyaby

We're a small team trying to build applications on MediaPipe. In first few projects, the rendering stuff is quite straightforward. We wrote OpenGLES directly in calculators pretty mush like MediaPipe built-in examples. However when things become complicated, we realize that we need a 3D rendering engine to go further.

You mentioned two ways to integrate Filament with MediaPipe. The first one is to integrate Filament as calculators, and it only support OpenGL backend and might cause more issues in the future. The second one is initialize Filament and MediaPipe separately and communicate through application level signals. It seems to me that the second one is reasonable and should be less buggy. However, I still have two questions about it.

1) How to balance GPU resource between the two components?

2) In realtime applications, we use FlowLimiterCalculator in the high level graph to control the number of in-flight streams. If Filament is a separate element of the system, it means we can't have a back edge signal representing the completion of specific timestamp. Is there any possible way to solve this?