Open khanhlvg opened 4 years ago
Summarizing the discussion between @khanhlvg and myself from today's meeting -
After sufficient data collection, we can utilize the data in order to perform pose classification. So, the problem statement becomes - given a set of body landmarks predict the pose.
I am happy to work on the model training part as well as the model conversion part for the pose classification part. Also, I can work with a mobile developer to help them develop the application such that we gather the data in an expected way.
Hi @khanhlvg . I can help with building an android app to collect the data required and once it is done can help with training a model too. I am creating a basic repo for the android app here https://github.com/Anil-matcha/pose_detection_data_collector . Kindly let me know if you have any other information to discuss
@Anil-matcha thank you for your interest! @sayakpaul is looking for someone who can develop the Android app for data collection so I think this can be a nice collaboration.
@sayakpaul WDYT?
Yeah exactly. @Anil-matcha and I have discussed about it and we are game :D
@Anil-matcha @sayakpaul I would like to jump in on the project, is there any thing either of you need help with? I can help with on device inference for Android.
@anilmaddala you can collaborate @Anil-matcha on the Android part for now on. Once that is sorted out we can work on the next steps together i.e. collecting the data and building the downstream model and so on.
Is there an update you, folks, would like to share? @anilmaddala @Anil-matcha
@sayakpaul I am working on the data collection part of the app. I have integrated the MLKit apis to get the Pose points, need to implement the persistance part. Can we do on device training from pose classification instead of doing the training on Firebase?
@anilmaddala thanks for the updates. Actually, after we collect the data, the plan is to train a shallow neural network externally with TensorFlow/Keras, convert it to TensorFlow Lite and then redeploy it.
But I am curious regarding the on-device model training part. Do you have a pipeline in mind?
@sayakpaul we might have to still build a base model in cloud. I was thinking for users to test out the app on new poses, we can implement something like https://blog.tensorflow.org/2019/12/example-on-device-model-personalization.html for on-device training and inference.
@anilmaddala so the idea here is after collecting the poses, we will build a shallow model, convert it to TFLite and deploy that TFLite model to the application to perform on-device inference.
Now coming to personalized training (as described in the blog post you mentioned), in order for that to work we still need a master model first which we could transfer learn for a particular user. Our initial shallow model can act as a master model here. Tagging @khanhlvg for any inputs he might have.
Let me know if there's any doubt in that regard.
Well, it this google-research/poem may help for it.
Goal:
Technologies used:
Artifacts:
data collection app