Face Masking

[enijar]

This is a bit of a complex question and maybe out of the scope of this project, so please close if so.

I'm trying to apply a "face mask" inside the area of the extended points (see reference image).

How can I apply effects to the face area, such as making the eyes reddish and cheeks more pink?

I've tries overlaying a PNG image but when the head is rotated the image is not rotated along with the head.

[MarcelKlammer]

You will need a snapshot of your tracked (extended) face and generate the UV data in that snapshot.

• look straight into the camera, most preferably with rotation x,y,z being 0 (log that continuously).
• take snapshot and "save" the BRFFace vertices (including the extended ones)
• crop the image to the bounds of the extended shape

This cropped area has a width and height. Now transform the shape vertices into uv data by dividing x values by width and y values by height.

Save the cropped facial image and the uv data to your disk.

Now you can open this image in Photoshop and paint on a layer to export that layer as transparent png.

Then you need to combine the face texture example and the extended face example and include your uv data and texture image (exported png).

Thats all :D

[enijar]

@MarcelKlammer Brilliant, thanks for the quick answer!

Is there an example of how to create the UV file that you can point me to?

[MarcelKlammer]

it's just a text file of the form x,y,x,y,x,y,...,x,y

[MarcelKlammer]

That's the one form the examples with only the normal 68 landmarks.

https://github.com/Tastenkunst/brfv4_javascript_examples/blob/master/assets/brfv4_face_textures.js

[enijar]

@MarcelKlammer Ah, silly me! Thanks for sharing the example.

[enijar]

Sorry if this seems like a waste of your time but I have been trying to apply my own custom texture with the steps you outlined and can't seem to get it working.

The example from face_texture_overlay.js works perfectly in my integration. However, when I try to apply my image texture it doesn't map and I can't see anything. Is there something I'm missing? Let me know if you need any more information.

Cropped image

UV Map Generation

// Divide "x" (i % 2 === 0) by "width" and "y" by "height" to generate the UV map.
extendedShape.vertices.map((v, i) => v / (i % 2 === 0 ? faceBounds.width : faceBounds.height))

UV map

[
  1.468952024559414,
  0.9659869985787226,
  1.4768002948167158,
  1.0597382177667902,
  1.4969832079833947,
  1.15434071647113,
  1.515206620263534,
  1.2519415247570016,
  1.5502804072966416,
  1.3481891289039525,
  1.6147766071725767,
  1.4270411363110362,
  1.7124641835875274,
  1.4957512612795678,
  1.8250840609489611,
  1.5504832190568774,
  1.9573454175525458,
  1.563029547472266,
  2.095395217946067,
  1.551537902814927,
  2.2173896832881717,
  1.4997933993325727,
  2.323857279026154,
  1.435000179476778,
  2.3956889539554216,
  1.3512493930479006,
  2.433940918499605,
  1.2516984722429085,
  2.454823408965973,
  1.1524136383406003,
  2.463456272095944,
  1.0510661014163198,
  2.468952024559414,
  0.945538683442894,
  1.5466899402801624,
  0.9363418066534994,
  1.6115932062470595,
  0.8977709215309946,
  1.6956683845095861,
  0.8853103101651022,
  1.7912889297475973,
  0.8902215992427681,
  1.8792020568004892,
  0.9136073284500621,
  2.0165438143605714,
  0.9156798938970341,
  2.102988801719908,
  0.8971480001774734,
  2.194010040483263,
  0.8862103060894276,
  2.284953468383213,
  0.8990159699487751,
  2.3539630585721274,
  0.9346663858373984,
  1.9458956930966633,
  0.9762170471253282,
  1.9460803137795113,
  1.0341489315751087,
  1.9459879133792177,
  1.0888997537223846,
  1.9459124440464242,
  1.151962349658387,
  1.8518589231408786,
  1.2147742149507665,
  1.893234849756589,
  1.2279318813346094,
  1.9463150071939026,
  1.2374047074516876,
  1.9990423139798024,
  1.2275238814171612,
  2.046203802501619,
  1.2157296125510986,
  1.659463818308752,
  0.9898411598781289,
  1.7128496355497662,
  0.9700909588125998,
  1.7753522922864218,
  0.9685718805995183,
  1.8378713397373596,
  0.9939502829922073,
  1.7767624340679016,
  1.006448027381086,
  1.7110136954324302,
  1.0074571056973576,
  2.0604023037779338,
  0.9953927784823026,
  2.12115763832502,
  0.9701646953323981,
  2.182847063468448,
  0.9727221081828917,
  2.2418727273644237,
  0.9911790739487615,
  2.1860360480438854,
  1.010415171289628,
  2.1260426115343076,
  1.0104013374183194,
  1.7936706266152058,
  1.3461921533183896,
  1.8547132489565779,
  1.3197539704782213,
  1.9039046644010256,
  1.3070874396212222,
  1.9471444493836731,
  1.3162285175899795,
  1.9915099702373447,
  1.3065400419383384,
  2.046510903247235,
  1.3204326906523798,
  2.1091191089791574,
  1.3426389003059869,
  2.044552302949392,
  1.356871769587141,
  1.9878407917687635,
  1.364618605138454,
  1.9467424265894129,
  1.3633701148005393,
  1.9031982426272405,
  1.3610558206549104,
  1.8554573153378902,
  1.356719795575061,
  1.8169004114609637,
  1.3443580070314838,
  1.9066505593376277,
  1.3341384166268244,
  1.9469717164716231,
  1.336825364817754,
  1.9895517301749808,
  1.3329387750785069,
  2.086323227767527,
  1.340928927907,
  1.9893031676946586,
  1.3303872532067045,
  1.9454317098000622,
  1.3351953508711158,
  1.9050100469675018,
  1.3282020310661087,
  1.563251056578139,
  0.6983223823929237,
  1.7014837584933376,
  0.593433247513603,
  1.8465049817315449,
  0.563029547472266,
  2.069035945772119,
  0.5644720429623612,
  2.2133049757866576,
  0.5967509871408221,
  2.374522953831457,
  0.7035812283161162
]

[MarcelKlammer]

x and y are not cropped to the bounds. I wasn't clear enough on that one.

(x - boundsX)/boundsWidth (y - boundsY)/boundsHeight

UV coords are a percentage of the texture image, ranging from 0.0 to 1.0

[enijar]

Ah, yes that works now but it's still not mapping to the extended face. I'm using the code from the example.

This is the mapped image in black.

This is the bounds of the extended face.

{
  "height": 230.87443063835497,
  "width": 170.55010986328125,
  "x": 196.54815673828125,
  "y": 88.09190847297316
}

What am I still missing?

[MarcelKlammer]

You need to extract the vertices of the extended shape and have the those converted to UV coords. Then you need to save the cropped-to-extended-shape-bounds texture. The drawing gets the vertices and the triangles of the extended shape.

[enijar]

Ah, that's what I was missing. I was drawing the UV map with the default face shape and not the extended face shape.

Thanks.

[sjdaley]

This is exactly what I've been looking for. @Enijar is there any chance I could see your project? I guessed it would be the extended face.js that would solve this. @MarcelKlammer thanks for your email response leading me here.

[enijar]

@sjdaley The whole project is growing but you can view the source here.

This is built for a 32" touchscreen (1920x1080) so I have two canvas elements: one in emory to process the face (854x480) and the other to render the camera and UV texture (1920x1080).

The main processing script can be found here.

[sjdaley]

@Enijar Thanks so much!! I'll check it out now.

[emomooney]

@Enijar Hey.. any chance of being able to take a look at what you managed to acheive? I'm looking into this and am gonna spend a bit of time trying to work it out myself, but I figured I would ask anyway, your links to that repo aren't available anymore.

[SurrealNautilus]

Hi! I would like to know a bit more about how to integrate a texture in "extended_face_shape.js" in the HTML5 / Browser - Javascript version. @MarcelKlammer when you say: "... extract the vertices of the extended shape and have the those converted to UV coords ...". How could I do it? My knowledge is basic and I have problems with that. Sorry!