FaceFilter Jumpy behavior

[ranbuch]

Hi, love this repo!

When compering your glasses VTO demo to karuneyewear.com or visagetechnologies.com it looks like the other solutions are more stable in terms of how steddy the glasses are, way less jumpy.

Can we solve it? Maybe by replacing the neural network to a haftier one or something like that?

Thanks.

[ranbuch]

And another thing, they also look more responsive, a bit more agile. It's like the video and the neural network are having different frame rate, neural network is lagging behind.

[xavierjs]

Hi @ranbuch I am sorry for my previous answer it was a bad joke. Jeeliz FaceFilter framerate depends on the device. It may lag on low end devices. I am regularly updating neural networks

[ranbuch]

@xavierjs No worries, thanks for taking the time to answer.

It may lag on low end devices

Right, I guess my question is, are you trying to reduce the weight of the neural network on expense of acuridy or preformance? I see you have multiple neural networks, is there something better in turms of acuridy than NN_DEFAULT.json?

[xavierjs]

Hi,

The issue is more because of the stabilization Settings (look for JEELIZFACEFILTER.set_stabilizationSettings in https://github.com/jeeliz/jeelizFaceFilter/#miscellaneous-methods). I have upgraded Jeeliz FaceFilter to set more stables values (indeed former ones were tuned for older neural networks). I hope it will work better. I will also update the online demos.

For VTO I have other solutions which may be more suited:

• Jeeliz VTO Widget : https://github.com/jeeliz/jeelizGlassesVTOWidget
• WebAR.rocks.face ( https://github.com/WebAR-rocks/WebAR.rocks.face - demo here: https://webar.rocks/demos/face/demos/VTOGlasses/ )

[ranbuch]

@xavierjs Thanks, appreciate it! Can you please share more light about the difference between those 2 other repositories and this one?

[xavierjs]

Hi,

Jeeliz VTO widget:

• is only for glasses VTO. It is more specialized. The solution embed its own 3D engine (THREE.js is not used) and some optimization are possible since glasses pose data does not go through the CPU but stays on the GPU between the output of the neural network and the input of the 3D engine.
• also does lighting reconstruction (both ambient and directional lighting) to light the glasses with a coherent lighting.
• glasses models can be provided:
  1. using a webapp called jeeliz glasses studio, to convert GLTF models -> legacy JSON format
  2. using a specialized CMS to handle materials, glasses mesh and models, very efficient for clients having important collection of glasses (very often a glasses mesh is sold with 10 combination of materials, so it is important to separate material, meshes and models to provide an efficient workflow). I provide both turnkey solution (where I use the CMS, input models from lightbox pictures) or the CMS as a service.

WebAR.rocks.face is closer to Jeeliz FaceFilter than Jeeliz VTO widget. It is like Jeeliz FaceFilter except:

• It provides specific landmark positions (for example around the mouth, the ears, ...) depending on the neural network used. Jeeliz FaceFilter only returns a 3D pose. This feature enable multiple new use-cases: flexible masks, makeup virtual try-on, earrings virtual try-on,
• The pose is more stable. Indeed with Jeeliz FaceFilter, the neural network directly outputs a 3D pose, which is harder to stabilize. In WebAR.rocks.face, the neural network outputs 2D landmark position, which is easier to stabilize since it is 2D data.Then the 3D pose is computed from 2D point positions (solvePnP).

[ranbuch]

Gav it, well ditching THREE.js is not an option for me.

Is there a way to configure jeelizFaceFilter to run like WebAR.rocks.face?

Is there a way to lower the stability for higher speed or vice versa?

[xavierjs]

Since jeelizFaceFilter and WebAR.rocks.face does not have the same pose computation pipeline it is not possible to use the same stabilization methods.

Did you try Jeeliz FaceFilter since the latest commit? ( https://github.com/jeeliz/jeelizFaceFilter/commit/581273a6277026378d4e6e7d081ac14b9c04d85a )

You can adjust the stabilization settings as described in the README https://github.com/jeeliz/jeelizFaceFilter/#miscellaneous-methods (search for JEELIZFACEFILTER.set_stabilizationSettings )

[ranbuch]

Did you try Jeeliz FaceFilter since the latest commit? ( 581273a )

Yes, I've experienced an indentation to one of the sides (left or right) but in terms of preference there was no change.

Can I use WebAR.rocks.face with three.js?

Thanks.

[xavierjs]

yes

[ghost]

We cannot see the source code, right?

[ranbuch]

@xavierjs The VTOGlasses.js example on WebAR.rocks.face requires React. Do you happen to have an independent example?

[xavierjs]

It is the Github repo of Jeeliz FaceFilter here dude