Face tracker

[calcoloergosum]

• Copy pasted hand_tracker.py, and changed keywords "HAND" into "FACE".
• Borrowed mediapipe landmark names from tfjs-models, and used it for face landmark coloring.

[mprib]

Hi @calcoloergosum,

Thanks for your PR. I'm starting to take a look and have run into some issues that I think may be related to the sample footage I'm using. I'm going to take some more close-up face footage as time permits. In the meantime, I just wanted to check in and see if you had confirmed that you're able to get 3D results with this face tracker.

The code looks good to me. I'm just wondering if this is more about the specific image needs of mediapipe and I want to make sure I'm testing against appropriate footage.

Thanks,

Mac

[calcoloergosum]

Forgive me for my late reply.

I just wanted to check in and see if you had confirmed that you're able to get 3D results with this face tracker.

In my case I used 3 cameras: One on laptop, two webcams on tripods. No hassle, just worked. But it was just white dots with no colors.

---

I hope some more context helps.

My use case is using your rigmarole for motion capture to make blender animation with media pipe dense landmarks. For example, I am using the following metarig point mapping (hand-picked by me):

{
    # forehead
    "forehead.L.head":      [10, 338],
    "forehead.L.tail":      [336, 285],
    "forehead.L.001.head":  [297],
    "forehead.L.001.tail":  [296, 334, 282, 295],
    "forehead.L.002.head":  [332, 284],
    "forehead.L.002.tail":  [293, 283, 300, 276],
    "forehead.R.head":      [10, 109],
    "forehead.R.tail":      [107, 55],
    "forehead.R.001.head":  [67],
    "forehead.R.001.tail":  [66, 105, 52, 65],
    "forehead.R.002.head":  [103, 54],
    "forehead.R.002.tail":  [63, 53, 70, 46],

    # brows
    "brow.T.L.head":        [353, 383, 265, 372],
    "brow.T.L.tail":        [293, 283, 300, 276],
    "brow.T.L.001.head":    [293, 283, 300, 276],
    "brow.T.L.001.tail":    [296, 334, 282, 295],
    "brow.T.L.002.head":    [296, 334, 282, 295],
    "brow.T.L.002.tail":    [336, 285],
    "brow.T.L.003.head":    [336, 285],
    "brow.T.L.003.tail":    [8],
    "brow.T.R.head":        [124, 156, 35, 143],
    "brow.T.R.tail":        [63, 53, 70, 46],
    "brow.T.R.001.head":    [63, 53, 70, 46],
    "brow.T.R.001.tail":    [66, 105, 52, 65],
    "brow.T.R.002.head":    [66, 105, 52, 65],
    "brow.T.R.002.tail":    [107, 55],
    "brow.T.R.003.head":    [107, 55],
    "brow.T.R.003.tail":    [8],

    "brow.B.L.head":        [342],
    "brow.B.L.tail":        [260],
    "brow.B.L.001.head":    [260],
    "brow.B.L.001.tail":    [257],
    "brow.B.L.002.head":    [257],
    "brow.B.L.002.tail":    [286],
    "brow.B.L.003.head":    [286],
    "brow.B.L.003.tail":    [414],
    "brow.B.R.head":        [113],
    "brow.B.R.tail":        [30],
    "brow.B.R.001.head":    [30],
    "brow.B.R.001.tail":    [27],
    "brow.B.R.002.head":    [27],
    "brow.B.R.002.tail":    [56],
    "brow.B.R.003.head":    [56],
    "brow.B.R.003.tail":    [190],

    # eyes
    "lid.T.L.head":         [263, 359],
    "lid.T.L.tail":         [388],
    "lid.T.L.001.head":     [388],
    "lid.T.L.001.tail":     [386],
    "lid.T.L.002.head":     [386],
    "lid.T.L.002.tail":     [384],
    "lid.T.L.003.head":     [384],
    "lid.T.L.003.tail":     [362],
    "lid.B.L.head":         [362],
    "lid.B.L.tail":         [381, 380, 256, 252],
    "lid.B.L.001.head":     [381, 380, 256, 252],
    "lid.B.L.001.tail":     [374, 373, 253, 254],
    "lid.B.L.002.head":     [374, 373, 253, 254],
    "lid.B.L.002.tail":     [390, 249, 255, 339],
    "lid.B.L.003.head":     [390, 249, 255, 339],
    "lid.B.L.003.tail":     [263, 359],

    "lid.T.R.head":         [33, 130],
    "lid.T.R.tail":         [161],
    "lid.T.R.001.head":     [161],
    "lid.T.R.001.tail":     [159],
    "lid.T.R.002.head":     [159],
    "lid.T.R.002.tail":     [157],
    "lid.T.R.003.head":     [157],
    "lid.T.R.003.tail":     [133],
    "lid.B.R.head":         [133],
    "lid.B.R.tail":         [154, 153, 26, 22],
    "lid.B.R.001.head":     [154, 153, 26, 22],
    "lid.B.R.001.tail":     [145, 144, 23, 24],
    "lid.B.R.002.head":     [145, 144, 23, 24],
    "lid.B.R.002.tail":     [163, 7, 25, 110],
    "lid.B.R.003.head":     [163, 7, 25, 110],
    "lid.B.R.003.tail":     [33, 130],

    "eye.L":                [473],
    "eye.R":                [468],

    # nose
    "nose.head":            [8],
    "nose.tail":            [5],
    "nose.001.head":        [5],
    "nose.001.tail":        [1],
    "nose.002.head":        [1],
    "nose.002.tail":        [94],
    "nose.003.head":        [94],
    "nose.003.tail":        [2],
    "nose.004.head":        [2],
    "nose.004.tail":        [0],
    "nose.L.head":          [357],
    "nose.L.tail":          [278],
    "nose.L.001.head":      [278],
    "nose.L.001.tail":      [1],
    "nose.R.head":          [128],
    "nose.R.tail":          [48],
    "nose.R.001.head":      [48],
    "nose.R.001.tail":      [1],

    # cheek
    "cheek.B.L.head":       [308],
    "cheek.B.L.tail":       [280, 411],
    "cheek.B.L.001.head":   [280, 411],
    "cheek.B.L.001.tail":   [353, 383, 265, 372],
    "cheek.T.L.head":       [353, 383, 265, 372],
    "cheek.T.L.tail":       [330],
    "cheek.T.L.001.head":   [330],
    "cheek.T.L.001.tail":   [357],
    "cheek.B.R.head":       [78],
    "cheek.B.R.tail":       [50, 187],
    "cheek.B.R.001.head":   [50, 187],
    "cheek.B.R.001.tail":   [124, 156, 35, 143],
    "cheek.T.R.head":       [124, 156, 35, 143],
    "cheek.T.R.tail":       [101],
    "cheek.T.R.001.head":   [101],
    "cheek.T.R.001.tail":   [128],

    # chin
    "chin.head":            [175],
    "chin.tail":            [18],
    "chin.001.head":        [18],
    "chin.001.tail":        [17],

    # lip
    "lip.T.L.head":         [11],
    "lip.T.L.tail":         [303],
    "lip.T.L.001.head":     [303],
    "lip.T.L.001.tail":     [308],
    "lip.B.L.head":         [15],
    "lip.B.L.tail":         [403],
    "lip.B.L.001.head":     [403],
    "lip.B.L.001.tail":     [308],
    "lip.T.R.head":         [11],
    "lip.T.R.tail":         [73],
    "lip.T.R.001.head":     [73],
    "lip.T.R.001.tail":     [78],
    "lip.B.R.head":         [15],
    "lip.B.R.tail":         [179],
    "lip.B.R.001.head":     [179],
    "lip.B.R.001.tail":     [78],

    # Outline
    "temple.L.head":        [251],
    "temple.L.tail":        [372, 345, 447, 264],
    "jaw.L.head":           [372, 345, 447, 264],
    "jaw.L.tail":           [435],
    "jaw.L.001.head":       [435],
    "jaw.L.001.tail":       [369],
    "chin.L.head":          [369],
    "chin.L.tail":          [308],

    "temple.R.head":        [21],
    "temple.R.tail":        [143, 116, 227, 34],
    "jaw.R.head":           [143, 116, 227, 34],
    "jaw.R.tail":           [215],
    "jaw.R.001.head":       [215],
    "jaw.R.001.tail":       [140],
    "chin.R.head":          [140],
    "chin.R.tail":          [78],
}

[mprib]

Thank you for your reply and the context! I will plan to record some test footage by the end of this week that has a single face more prominently in frame just to formally confirm everything is flowing through (though it seems like it is).

I really appreciate you taking the time to do this and hope that you are finding value in this tool! If you have any questions along the way, please don't hesitate to ask. There may be some delay in my responses due to outside constraints, but this project is still very much alive.

Mac

[mprib]

@calcoloergosum,

Thank you again for your submission and for your patience while I finally had a chance to block out some time to take a look at this.

Best,

Mac