Improve scaling in examples

[johanwitters]

Hi

This is a brilliant project. Thanks for the great work.

I was trying some examples, including https://github.com/jeeliz/jeelizFaceFilter/tree/master/demos/CSS3D/comedy-glasses and https://github.com/jeeliz/jeelizFaceFilter/tree/master/demos/threejs/miel_pops and I'm unsure how to adjust it so that the scaling is more accurate: moving your face away or closer to your camera causes the glasses to be too small or too big for your face. I've understood slightly how I can scale, but I'm far from the point of improving these 2 examples to make them scale correct. Anyone thoughts on how to improve the scaling in these examples?

Thanks 1000x

Johan

[xavierjs]

Hi Johan,

You can change the scale of the 3D object (for the miel pops demo). Then it may be a problem with the FoV of the camera. This information is not available in the browser so we have to guess it (we have a value for the mobile and a value for desktop).

Is the scaling good in our VTO demo ? https://jeeliz.com/sunglasses

Best, Xavier

[johanwitters]

Yes indeed: The scaling is correct in your VTO demo, but I'm unsure where to locate the code in that demo to reuse and fix / improve the miel pops demo

[xavierjs]

What do you mean exactly by the scaling ?

• The size of the 3D model
• The translation movement of the 3D model when the head rotates (along which axis?)
• The scale of the glasses when your head rotates ?

I have retried the miel pops demo. The glasses are a bit too small, but their relative scale is the same if I am close or far from the camera.

Many tweaks have been applied for the VTO demo to better positionnate and scale the 3D glasses using global face detection (like FaceFilter lib). Most are empirical results like increase the Scale of the 3D model when the user rotates his head along the Y axis, etc... But on the VTO demo the workflow is really different: it uses a custom WebGL 3D engines with a 100% GPU workflow (on the CPU side the information if the user is detected is even not available). I explain the main reasons why we have not released the code of the VTO demo publicly here: https://github.com/jeeliz/jeelizGlassesVTOWidget/issues/2.

[johanwitters]

Hi,

When I say scaling I mean, when close or far from the camera, the glasses should be relative scaled with respect to my face.

So, using the mielpops example, I took a photo of my own pair of glasses, then created a model of it and used it to replace the mielpops example model. When using this model, I basically have the same glasses put twice on my nose: 1 real, 1 virtual When I do so, the issue is quite clear: when I move my face about .5 meter away, perhaps a bit more, from my camera, the size of the glasses overlaps my real glasses perfectly. In any other case, the virtual glasses are far different. See attached photos - 50 cm and 20 cm - please.

I also notice this behaviour with the model provided in the example

thanks,

Johan

[xavierjs]

Hi @johanwitters

Thank you for these detailed explanations. The problem comes from the FoV of the camera. On the second picture, you have a strong fisheye effect because you are close to the camera. But the glasses are rendered without this fisheye effect, with a camera with a smaller FoV than the real camera.

In the miel pops demo script, https://github.com/jeeliz/jeelizFaceFilter/blob/master/demos/threejs/miel_pops/demo.js the FoV is 40° (cf line 6).

This is a smaller value than the real webcam FoV. You can try to increase it (for example to 70°). Unfortunately there is no way to get the real webcam FoV in the browser, so we have to make hypothesis, depending it is a desktop/mobile device, in landscape/portrait mode.

I did a small draft:

C2 is the current THREE.js camera, with a small FoV. The position of the camera is computed using the middle of the user head (L).

If you increase the FoV, the computed position will be closer. It will increase the fisheye effect. It is the C1 camera. On the C1 camera the glasses looks larger (alpha1 > alpha2)

But if C1 and C2 are far away (like in your first picture), it won't change anything. The fisheye effect will disappear.

If you want to do glasses VTO, we can discuss about it. Although our current VTO solution is not very open, we can discuss that. I think we can provide a very good solution for a very honest price. Our VTO app also reconstructs the lighting (both ambient and directionnal) and we have an image based server side fallback for people who are not compatible with WebGL.

[xavierjs]

I am going to push some modifications in 5 minutes...

[johanwitters]

Hi,

that's great. Thanks for the comprehensive explain and update. I've tried it and with the FoV of 60° it's much better indeed.

I tried to change the comedyGlasses behaviour, changing the "model", being the png in this case, and the cameraFOV. That's not working (yet). The glasses are way too big. I'll guess I need to play a bit with the other parameters. Initial attempts are fruitless. You wouldn't have tips to change the cameraFOV in this example. For example, what are, in this example, the SETTINGS positionOffset, scale, pivotOffsetYZ. What's the purpose of these arrays?

many thanks!

Johan

[xavierjs]

Hi,

I have not forgotten this issue, I really need to spend a couple of hours because I think I did a mistake in the computation of the unprojection params.

scale is the global scale of the 3D model following the head, and pivotOffsetYZ is the YZ of the 3D position of the rotation pivot point (the X is always = 0). PositionOffset is only a 3D translation factor.

Since the neural net learns to center the detection frame according to the point above the nose and between the 2 eyes, it should be possible to get a better positionning.

New information will be posted here soon...

[johanwitters]

Great. Thanks very much.

[xavierjs]

Hi,

I have improved the tracking in the helper (https://github.com/jeeliz/jeelizFaceFilter/blob/master/helpers/JeelizThreejsHelper.js) by adding a new parameter, _settings.tweakMoveYRotate .

I also have added a flag (still in the helper) to display a cube in the position of the pivot point, isDebugPivotPoint in order to better adjust _settings.pivotOffsetYZ

I have added a simple VTO demo, https://jeeliz.com/demos/faceFilter/demos/threejs/VTO/ The code is here: https://github.com/jeeliz/jeelizFaceFilter/tree/master/demos/threejs/VTO

Does it work better for you ?

[johanwitters]

That's much much better. This looks perfect. I'll post some more comments in a bit, when I have the time to thoroughly test it. But thanks very much already.

[xavierjs]

Hi,

Thank you for your feedback. Since the output of the neural net is the global 3D transforms and not specific keypoints, I don't think it will be possible to get a perfect result, especially for the end of the glasses branches. https://jeeliz.com/sunglasses should be a bit better because the neural network is more specialized and we have added some tweaks but I think this is close to the limit of what we can do without keypoints.

I am working on a face landmarks based version of FaceFilter, where the neuron network outputs landmarks position which are then used to compute the glasses 3D transforms using OpenCV solvePnP method.