Working on morph targets over virtual avatar based on WebAR.rocks face landmarks

[Juanmaramon]

Hi @xavierjs,

I'm trying the face API and currently working on a basic face tracking demo (I sent you auth data on an email).

Here is the avatar glb file: https://juanmaramon.com/Test/accdemo/demos/xrs/assets/avatar.glb

Based on the Flexible mask demo I was trying to load the GLB file using

`_flexibleMaskHelper.load_geometryFromGLTF(threeLoadingManager, _spec.flexibleMaskURL).then(function(geom){ const face3DKeypoints = (_spec.flexibleMaskPoints) ? _spec.flexibleMaskPoints : WebARRocksFaceHelper.get_facePointPositions();

_flexibleMaskMesh = _flexibleMaskHelper.build_flexibleMask(geom, face3DKeypoints, {
  kpInfluenceDecay: _spec.kpInfluenceDecay // [ distance from the keypoint where decay start, distance from the keypoint where decay ends ]
});`

But it raise this error:

Error: MULTIPLE_GEOMETRIES

Does it need to be all geometry grouped on the root object?

Can you explain some of the options on the settings array?

flexibleMaskPoints and kpInfluenceDecay:

https://github.com/xavierjs/WebAR.rocks.face/blob/69df4f951a82041a121a8bac5ead2fbc3b4bc275/demos/flexibleMask/main.js#L13

Thanks!

[xavierjs]

Hi @Juanmaramon

This error is triggered by the helper by this function: https://github.com/xavierjs/WebAR.rocks.face/blob/69df4f951a82041a121a8bac5ead2fbc3b4bc275/helpers/WebARRocksFaceFlexibleMaskHelper.js#L158

I the last commit I have added an extra argument to pick the geometry you want to use by specifying the name:

_flexibleMaskHelper.load_geometryFromGLTF(threeLoadingManager, _spec.flexibleMaskURL, <geomName>)

in your case it would be:

_flexibleMaskHelper.load_geometryFromGLTF(threeLoadingManager, _spec.flexibleMaskURL, Wolf3D_Head)

All geometries should be grouped in the root object. But only one geometry will be deformed. I advise you to start from fflexibleMask2 demo instead of flexibleMask since you case is closer to flexibleMask2 demo.

Best regards, Xavier

[Juanmaramon]

Hi @xavierjs !

Thanks for the info, now based on FlexibleMask2 example I render some avatar face elements :)

I'm using the default values on foolMaskARMetadata.json only chaging the DEFORMEDID value to

"DEFORMEDID": "Wolf3D_Head",

then is the DEFORMEDKEYPOINTS array with key value pairs:

{
 "label": "LEFT_EYEBROW_INSIDE",
 "co": [18.183937,53.444672,41.949951]
},

with the current glb avatar file that I'm using how can I achive the same effect as the FlexibleMask2 demo?

Thanks!

[xavierjs]

Hi,

You need to provide more points so that the helper can compute the mapping between landmarks and mesh keypoints. I also advise you to align your mesh with the mesh provided as an example (https://github.com/xavierjs/WebAR.rocks.face/blob/master/demos/flexibleMask2/assets/face.glb) to make debug easier. it would also be better to apply visual transforms before exporting and to triangulate faces.

Best regards, Xavier

[Juanmaramon]

Hi @xavierjs ,

Aligning with the mesh (https://github.com/xavierjs/WebAR.rocks.face/blob/master/demos/flexibleMask2/assets/face.glb) rotates 90º the geometry and resulting in this:

What are you meaning with

it would also be better to apply visual transforms before exporting and to triangulate faces."

Right now, the avatar is rendered at 1:1 scale and without rotations applied to it.

DEFORMEDKEYPOINTS are 2D coordinates based on which origin?

{ "label": "LEFT_EYEBROW_INSIDE", "co": [18.183937,53.444672,41.949951] },

Thanks!

[xavierjs]

I opened Blender then :

• Created a new file
• Imported assets/face.glb, the face base model (There was a bug, it was wrongly 90° rotated around X axis, I have fixed this in the latest commit)
• Imported avater.glb, your model
• aligned avatar.glb on face.glb (see screenshot1.png)
• I have selected AvatarRoot object and:
  • Changed the origin of avatar.glb (set Origin to Geometry)
  • Done object/apply/all transforms
• I have selected Wolf3D_Head, which will be the flexible mask and I have:
  • Done object/apply/all transforms
• In OBJECT mode, I selected AvatarRoot and all its children from the graph view (select Hierarchy)
• I exported it as GLB, named it avatar2.glb and checked Include/Selected objects, Geometry/Apply modifiers, in animation section: unchecked animation, skinning, shape keys as export options,
• I saved the blender file as avatar.blend

Then I:

• Edited main.js script and replace assets/foolMask.glb by assets/avatar2.glb

• Edited foolMaskARMetadata.json and set:

  "ID": "AvatarRoot",
  "DEFORMEDID": "Wolf3D_Head",

  Now it works but since the keypoints are not very well positionned, the result is not great. I go back to Blender,

  • Now I have copy/cut the python script given at the end of this post into the Python Blender console
  • I have selected Wolf3D_Head and entered into EDIT mode
  • I selected the vertice LEFT_EYE_INSIDE and I run logSelected (see screenshot2.png)
  • I got the coordinate of the point and I copy/paste it to foolMaskARMetadata.json

I also removed the ear points from foolMaskARMetadata.json. If you want a very good result you have to merge the eyes, the hair to the geometry. Otherwise when the skin moves the eyes are still static and there are intersection bugs. You also need to find the right position for all of the landmarks (not only LEFT_EYE_INSIDE)

I can finish the demo as a development service if you are interested.

import bmesh

def logSelected():
  obj = bpy.context.object
  me = obj.data
  bm = bmesh.from_edit_mesh(me)
  vertices = [e for e in bm.verts]
  for vert in vertices:
    if vert.select:
      co = vert.co
      print('[%f,%f,%f]' % (co.x, co.z, -co.y))