ml-gde / e2e-tflite-tutorials

Project tracking of the "Mobile ML Working Group", for the End-to-End TensorFlow Lite tutorials.
Apache License 2.0
133 stars 26 forks source link

Custom pose classification tutorial #33

Open khanhlvg opened 4 years ago

khanhlvg commented 4 years ago

Goal:

Technologies used:

Artifacts:

sayakpaul commented 4 years ago

Summarizing the discussion between @khanhlvg and myself from today's meeting -

After sufficient data collection, we can utilize the data in order to perform pose classification. So, the problem statement becomes - given a set of body landmarks predict the pose.

I am happy to work on the model training part as well as the model conversion part for the pose classification part. Also, I can work with a mobile developer to help them develop the application such that we gather the data in an expected way.

Anil-matcha commented 4 years ago

Hi @khanhlvg . I can help with building an android app to collect the data required and once it is done can help with training a model too. I am creating a basic repo for the android app here https://github.com/Anil-matcha/pose_detection_data_collector . Kindly let me know if you have any other information to discuss

khanhlvg commented 4 years ago

@Anil-matcha thank you for your interest! @sayakpaul is looking for someone who can develop the Android app for data collection so I think this can be a nice collaboration.

@sayakpaul WDYT?

sayakpaul commented 4 years ago

Yeah exactly. @Anil-matcha and I have discussed about it and we are game :D

anilmaddala commented 4 years ago

@Anil-matcha @sayakpaul I would like to jump in on the project, is there any thing either of you need help with? I can help with on device inference for Android.

sayakpaul commented 4 years ago

@anilmaddala you can collaborate @Anil-matcha on the Android part for now on. Once that is sorted out we can work on the next steps together i.e. collecting the data and building the downstream model and so on.

sayakpaul commented 3 years ago

Is there an update you, folks, would like to share? @anilmaddala @Anil-matcha

anilmaddala commented 3 years ago

@sayakpaul I am working on the data collection part of the app. I have integrated the MLKit apis to get the Pose points, need to implement the persistance part. Can we do on device training from pose classification instead of doing the training on Firebase?

sayakpaul commented 3 years ago

@anilmaddala thanks for the updates. Actually, after we collect the data, the plan is to train a shallow neural network externally with TensorFlow/Keras, convert it to TensorFlow Lite and then redeploy it.

But I am curious regarding the on-device model training part. Do you have a pipeline in mind?

anilmaddala commented 3 years ago

@sayakpaul we might have to still build a base model in cloud. I was thinking for users to test out the app on new poses, we can implement something like https://blog.tensorflow.org/2019/12/example-on-device-model-personalization.html for on-device training and inference.

sayakpaul commented 3 years ago

@anilmaddala so the idea here is after collecting the poses, we will build a shallow model, convert it to TFLite and deploy that TFLite model to the application to perform on-device inference.

Now coming to personalized training (as described in the blog post you mentioned), in order for that to work we still need a master model first which we could transfer learn for a particular user. Our initial shallow model can act as a master model here. Tagging @khanhlvg for any inputs he might have.

Let me know if there's any doubt in that regard.

SunAriesCN commented 3 years ago

Well, it this google-research/poem may help for it.