mit-cml / appinventor-extensions

Source code of extensions published for MIT App Inventor
Apache License 2.0
87 stars 96 forks source link

expose posenet {Part}Confidence and PoseConfidence #32

Open wr200m opened 4 years ago

wr200m commented 4 years ago

Describe the desired feature Currently only the configuration properties of MinPartConfidence and MinPoseConfidence can be queried? Would like to see PoseConfidence and {Part}Confidence to be available for all parts.

Give an example of how this feature would be used

// the following callback only works when the pose detection passes MinPoseConfidence 
// so I would want to paint the canvas whether there's a pose inside the camera feed or not,
// and meanwhile I want the pose analysis to start only when the picture has an actual pose in it.
when posenetExtension.PoseUpdated {
    canvas.setBackgroundImageInBase64(posenetExtension.backgroundImage);
    if (posenetExtension.PoseConfidence > 0.4) {
        // analyze the pose
    }
}

Same thing goes for {Part}Confidence properties but that can wait.

Why doesn't the current App Inventor system address this use case? As in the example, currently I have to set a low MinPoseConfidence so that I can paint the canvas's background image on every frame, that means I would have to compute inside the callback each time whether there's a real pose inside the frame or not. This is not good.

Why is this feature beneficial to App Inventor's educational mission?

ewpatton commented 4 years ago

Why not just increase the minimum confidence in this case?

wr200m commented 4 years ago

I would like to paint the canvas even when no body is detected in the frame. Currently as I understand, the only event for me to paint the canvas is when PoseUpdated() {}. With a high min confidence, this event would not trigger and I would not get a chance paint the canvas with nobody in it.