search

diablodale / dp.kinect2

Kinect v2 (XBox One) plugin for Cycling '74 Max
21 stars 4 forks source link

face hair/skin color output #37

Closed aartcore closed 7 years ago

aartcore commented 7 years ago

Hey Diablodale,

I'm trying to get the face skin/hair color out from the dp.kinect2 but i don't see/get any output of it? The pose, bounds and other face properties work like they should. The facecolors attribute is toggled on. Is there something i'm missing? I also can't find any documentation of this attribute how it should work.

Thank in advance,

Aart

diablodale commented 7 years ago

That is my oversight. I need to add docs on hair/skin color. Until then here is info...

Visit https://github.com/diablodale/dp.kinect2/wiki/Message-based-Data#face-tracking You will need to turn on @faces=1 and then enable, at least, @facecolors=1

Microsoft's builds a 3D model of the entire head. After that is built, it has data on the skin and hair colors. Therefore, the process is the same as @face3dmodel. What I write at the above link is true.

Some face features (shape units, hidef 2D points, and 3D model) require a fully captured/modeled face. This full capture is: a) drastically slow compared to the Kinect v1; b) large facial hair tends to interfere; and c) requires the face to be rotated slowly in four directions. The status of this capture can be seen with the modelstatus message.

Install the optional face data at the same download location as dp.kinect2. Enable the attributes, and then start slowly rotating your head in the four directions up down left right. And watch the modelstatus messages so you can see progress on the model being built. When it is complete, the skin and hair color will be available.

If you need faster results, it might be possible to get "good enough" answers by using the RGB image and then focusing on the face with the face's bounding box and using the 2D face points (which are fast) to find the colors of pixels. Its just an idea.

aartcore commented 7 years ago

Thanks it works! and indeed it takes a while before it outputs the data. Thanks for the idea to get faster the results. I was already setting up a same kind of system which you suggested to get some color output. I just tested two persons which had the exact same skincolor. Does the dp.kinect2 has a couple of skin profiles and output which it matches? When i get the chance i will test some more persons.

diablodale commented 7 years ago

The skin and hair color are solely determined by the Microsoft driver/APIs. I do not remember Microsoft releasing any information on their learning models and techniques which are used for skin and hair color.

Just now I did a quick search through the official Kinect forum and I do not see anything relevant on color models. I did find one example of a person with a brown pigmented skin tone who posted Kinect color output which seemed to be a reasonable match to his profile picture.

So with our sample set of less than 10...it seems to work. ;-)

diablodale commented 7 years ago

closing this issue as resolved. I have updated the wiki with this and more documentation on v1.1