Gesture Classifier Using Posenet, TensorFlow.js and ml5.js
:rainbow: Based on Daniel Shiffman's Color Classifier Example. :rainbow:
:pray: Thanks to Yining for Posenet + KNN Example.:pray:
Trains model based on recorded posenet data in JSON file. (Will upload the recording code soon).
User has two options:
- Train model with training data in JSON file, then test based on live input using Posenet.
// to test on live input set testLive on line 18 to true
let testLive = true;
- Train model with training data in JSON file, then test the model based on the recorded training data. Bad practice, but we're learning here!
// to test from recorded data set testLive on line 18 to false
let testLive = false;
The current poses are:
- T pose: Stand with legs together and arms out in a T
- Boss pose: Stand with legs together with hands on hips
- X pose: Stand with legs wide apart and arms over head and wide apart so you look like an X
I included two recordings:
- fullBodyLisa.json I recorded standing far away from camera. Legs are in full view.
- closeBodyLisa.json I recorded from about two feet from camera. Legs are not in full view.
Results
It works! It's not so bad considering my data is not so good. Posenet had a lot of trouble catching my arms overhead in pose 3, and you can see that in the training and testing results. I'll test with better data once I have it.
Issues
- I recorded data in my tiny apartment with bad light and it's not very good. Will record better data at ITP with more space. Maybe today!
- I'm getting an
identifier has already been used
error at the top of my sketch.js file. It's not affecting anything, but I would like to figure out where it's coming from. Filed issue here.
What I Want To Do Next
- Train from live data. I believe I just need to call model.fit each time a user adds data to live train it. Is this correct?
- Add the recording functionality to repo
- Record new data.