search

ml5js / ml5-library

Friendly machine learning for the web! 🤖
https://ml5js.org
Other
6.48k stars 902 forks source link

BodyPix #392

Closed joeyklee closed 5 years ago

joeyklee commented 5 years ago

→ Description 📝

Hi All! I just wanted to port over BodyPix (https://github.com/tensorflow/tfjs-models/tree/master/body-pix) to ml5.js this summer and figured I'd start an issue indicating I'd like to do so. I'm happy to discuss/collaborate if there's interest.

@oveddan: I know you've been a major part of developing BodyPix, so feel free to hop on and off as you'd like or if you feel the need or interest. Thanks for developing this wonderful functionality!

shiffman commented 5 years ago

Woohoo! Excited about this!

WenheLI commented 5 years ago

Interested in this feature! Just let me know if I can offer any help here!

joeyklee commented 5 years ago

Hi @WenheLI - Oh yes! I'd love your help on this. I will make a PR with at [In Development] tag so we can work together!

joeyklee commented 5 years ago

@WenheLI - I just made an "in progress" PR here: https://github.com/ml5js/ml5-library/pull/400 I'd be happy to hear your thoughts on the best way to structure things. The developers of BodyPix have done a really nice job wrapping up their functionality already, so maybe we can just sprinkle our own function names on top with some defaults so it is consistent with the ml5.js style and it might be cool?

WenheLI commented 5 years ago

@joeyklee - Yes, I see. They have done a great job and we just need to wrap them into an ml5 style class. In terms of the structure, I think we are pretty there! You PR has made a great infrastructure. For the next step, maybe we can some primary functions:

The above functions will give users access to the bodyPixel's basic functionality.

In addition, some helper functions might be good in this case? Like, return central points with bounding boxes for each body parts, or return an SVG mask instead of applying a mask directly onto the canvas which will give users more flexibility as an SVG mask can interact with other HTML Element like div, p or so(this might require a lot of extra work).

The above is still a rough idea! And it might be a little out of the scope for now.

joeyklee commented 5 years ago

@WenheLI - This is great! Thanks for this feedback

in segment() I might return maskPerson and maskBackground:


const result = {};
result.maskBackground = bp.toMaskImageData(segmentation, true);
result.maskPerson = bp.toMaskImageData(segmentation, false);

Maybe that makes it too slow and rather better to separate into different functions as you've mentioned above?

In addition, some helper functions might be good in this case? Like, return central points with bounding boxes for each body parts, or return an SVG mask instead of applying a mask directly onto the canvas which will give users more flexibility as an SVG mask can interact with other HTML Element like div, p or so(this might require a lot of extra work).

  • this would be really cool, but definitely tougher. It might be something to contribute directly to the tf/bodyPix lib so we can share the functionality :)

result.maskPerson

Screen Shot 2019-05-29 at 13 41 43

result.maskBackground

Screen Shot 2019-05-29 at 13 42 29

joeyklee commented 5 years ago

There's also these awesome functions like draw bokeh effect which we should remember to incorporate as well.

joeyklee commented 5 years ago

Part Segmentation

Added segmentWithParts(): Does this name make sense? Also, right now the code is redundant with the checking of the function inputs. We should optimize :)

If P5 exists then we get a P5.Image object, if not, then we get the ImageData object. Is there a better way to handle this, @WenheLI in your opinion? Happy to hear what would be best :) Thanks!

Screen Shot 2019-05-29 at 14 29 40

WenheLI commented 5 years ago

@joeyklee - That is great! By far, I think it is the best way to do so. Just one little comment for now, should we also return the raw data of the BodyPix for a broader use case?

A minor feature, is it possible for us to support p5Color?

joeyklee commented 5 years ago

@WenheLI - Thanks so much for the feedback!

Just one little comment for now, should we also return the raw data of the BodyPix for a broader use case?

  • Yes definitely good practice to return the raw data of bodyPix. The other models like UNET return blob, raw (which usually is the tensor), and image if p5 is available. Should our "raw" be the raw return of BodyPix or should we convert to tensor?

A minor feature, is it possible for us to support p5Color?

  • This is a nice idea! Are you thinking that if p5 is available to allow setting a color ramp?
  • Oh! This reminds me that actually we might also consider to do what we did with PoseNet Parts for the segmented parts. Though this implementation might be trickier since we have just the 1D array and the color values. I think the current idea is that people will use the color pixel values to do their own custom operations later on, but more intuitive would be to target the name of the body part.
WenheLI commented 5 years ago

Just one little comment for now, should we also return the raw data of the BodyPix for a broader use case?

  • Yes definitely good practice to return the raw data of bodyPix. The other models like UNET return blob, raw (which usually is the tensor), and image if p5 is available. Should our "raw" be the raw return of BodyPix or should we convert to tensor?

I think just the raw return of BodyPix will be fine, as the tensor import some extra complexity here.

  • This is a nice idea! Are you thinking that if p5 is available to allow setting a color ramp?

Yes, this is exactly what I was thinking!

  • Oh! This reminds me that actually we might also consider to do what we did with PoseNet Parts for the segmented parts. Though this implementation might be trickier since we have just the 1D array and the color values. I think the current idea is that people will use the color pixel values to do their own custom operations later on, but more intuitive would be to target the name of the body part.

We might need a map whose key is every body part and value is the corresponding color. And we can allow people to change the map to do the custom operations?

joeyklee commented 5 years ago

@WenheLI :

I think just the raw return of BodyPix will be fine, as the tensor import some extra complexity here.

We might need a map whose key is every body part and value is the corresponding color. And we can allow people to change the map to do the custom operations?


/* eslint class-methods-use-this: "off" */
    bodyPartsSpec(colorOptions){
        const result = {};

        const DEFAULT_COLOR = [
            [110, 64, 170], [106, 72, 183], [100, 81, 196], [92, 91, 206],
            [84, 101, 214], [75, 113, 221], [66, 125, 224], [56, 138, 226],
            [48, 150, 224], [40, 163, 220], [33, 176, 214], [29, 188, 205],
            [26, 199, 194], [26, 210, 182], [28, 219, 169], [33, 227, 155],
            [41, 234, 141], [51, 240, 128], [64, 243, 116], [79, 246, 105],
            [96, 247, 97],  [115, 246, 91], [134, 245, 88], [155, 243, 88]
          ];

        const bodyPartsIds = [-1,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23];
        const bodyPartsName = [
            "none","leftFace","rightFace","rightUpperLegFront","rightLowerLegBack",
            "rightUpperLegBack","leftLowerLegFront","leftUpperLegFront","leftUpperLegBack","leftLowerLegBack",
            "rightFeet","rightLowerLegFront","leftFeet","torsoFront","torsoBack","rightUpperArmFront","rightUpperArmBack",
            "rightLowerArmBack","leftLowerArmFront","leftUpperArmFront","leftUpperArmBack","leftLowerArmBack","rightHand",
            "rightLowerArmFront","leftHand"
        ];

        // TODO: allow for adding custom palettes
        const palette = colorOptions !== undefined ? colorOptions : DEFAULT_COLOR;
        // Add DEFAULT_COLOR as result.palette;
        result.palette = palette;
        // Iterate over the bodyPartsName
        bodyPartsName.forEach( (part, idx) => {
            result[part] = {id: bodyPartsIds[idx], color: DEFAULT_COLOR[idx]}
        })

        return result;
    }

the segmentation then returns:

  // // wrap up the final js result object
        const result = {};
        result.image = bp.toColoredPartImageData(segmentation, bodyPartsMeta.palette);
        result.raw = segmentation;
        result.bodyParts = bodyPartsMeta;

Where bodyParts is an object that looks like:

Screen Shot 2019-05-30 at 14 01 34

joeyklee commented 5 years ago

Update: https://github.com/ml5js/ml5-library/pull/400/commits/fef460339d92516a52f0dc613fc73b939db028b9

See working examples here: https://github.com/ml5js/ml5-examples/pull/153/commits/665ce7e20586d5ca8bbc16e4d46f6a28307daff8

.segment() and .segmentWithParts() take {option} that allows for setting segmentation parameters and setting of color palette.

if no option is specified, the defaults apply.

const options = {
    palette: [
        [127,103,14],[222,9,77],[17,223,92],[95,107,62],
        [112,173,9],[189,161,46],[239,196,5],[155,143,78],
        [134,165,84],[153,124,215],[111,60,183],[203,51,155],
        [68,245,152],[70,160,237],[195,148,225],[165,160,237],
        [4,24,186],[120,210,162],[141,253,217],[117,61,51],
        [244,184,186],[57,162,173],[7,252,62],[47,180,5]],
    outputStride: 8, // 8, 16, or 32, default is 16
    segmentationThreshold: 0.3 // 0 - 1, defaults to 0.5 
}

...

bodypix.segmentWithParts(gotResults, options)

Screen Shot 2019-05-31 at 10 39 55

joeyklee commented 5 years ago

Last notes: I changed the way that we define the color palette. This is cleaner and always ensures that colors are mapped to the desired part rather than counting by array index.

Screen Shot 2019-05-31 at 17 50 53 
function createSimplePalette() {
    options.palette = bodypix.config.palette;
    Object.keys(bodypix.palette).forEach(part => {
        const r = floor(random(255));
        const g = floor(random(255));
        const b = floor(random(255));
        options.palette[part].color = [r, g, b]
    });
}

function createHSBPalette() {
    colorMode(HSB);
    options.palette = bodypix.config.palette;
    Object.keys(options.palette).forEach(part => {
        const h = floor(random(360));
        const s = floor(random(100));
        const b = floor(random(100));
        const c = color(h, s, b)
        options.palette[part].color = c;
    });
}
joeyklee commented 5 years ago

🎉 BodyPix is added to ml5. Closed with: https://github.com/ml5js/ml5-library/pull/400