Pose Estimation from fingertips to enable markerless AR interaction.

[sudoankit]

It would be nice to have pose estimation from fingertips which would allow to use gestures and inspection of Augmented Reality Objects without markers in opencv_contrib.

In future this could be extended for speed, efficiency and more robust gesture detection in unfavorable conditions ( outdoor, variable light, automatic skin color estimation etc )

References

• Handy AR <— The main inspiration
• Initializing Markerless Tracking Using a Simple Hand Gesture (ISMAR, 2007)
• Realtime fingertip tracking and gesture recognition
• Statistical Color Models with Application to Skin Detection, CVPR
• https://www.researchgate.net/publication/313748784_Hand_Pose_Estimation_for_Markerless_Interaction_with_Augmented_Reality_Applications

For an exclusive list:

• https://github.com/xinghaochen/awesome-hand-pose-estimation

Also it gives some users the answers they are searching for.

• http://answers.opencv.org/question/100738/fingertip-pose-estimation/

[sudoankit]

Right, I'm working on this as I have experience on AR with OpenCV (and working on vision for a good time ) and if it's going good in 2 months of time, I'll apply for GSoC. The Handy AR paper is a good start to implement this. I was looking at the demo code given at the Handy AR website and though it's old (opencv2) it works alright.

This picture (taken from Handy AR paper gives a good pipeline which I would follow.)

I'll come back to this thread or elsewhere if you advice me to inform about my work in about a month which would be mostly:

• [ ] Convert opencv2 Handy AR implementation to opencv3 ( for *nix platforms ) (and if necessities arise, do those as well. )
• [ ] With results from the above decide on necessities and works required for the summer project which will go into the proposal.

Caveat:

• Some part of my undergraduate research relies on this module and my goal is to contribute to OpenCV (rather than GSoC) and if this project finishes before time, I will not apply for GSoC. (though it might be nice to have advanced gesture based interactions with AR objects through OpenCV )

[terfendail]

@sudoankit contributions to OpenCV are always welcome However I have a doubt about conversion of HandyAR implementation. I've quickly looked through HandyAR page and implementation and haven't found the license for the provided code. Could you please check the license for the implementation or ask authors to confirm they allow to use their code under BSD or BSD-compatible license so the contribution could be merged to OpenCV library

[sudoankit]

Yes, I have mailed the author, is there any messaging/slack group for discussing something further?

[terfendail]

There are an IRC channel #opencv and a Slack group open-cv intended for generic discussions related to OpenCV and a google group mailing list dedicated specifically to GSoC 2018

[sudoankit]

Thank you @terfendail.

So, I haven't got any replies from the author, thus I'm proceeding by writing my own demo code/ library by following the paper. Also, as the work (handy AR) is quite old, I have thought of some modifications for speed (especially in hand segmentation, where the proposed algorithm is taking time ).

Also, this module could be even used to power up a gesture engine for mobile devices allowing interaction with augmented reality objects in application developed using the OpenCV library in the future. What are your thoughts in this?

[terfendail]

@sudoankit There is a OpenPose project that could provide location for fingertips as a part of overall body pose estimation. While it have too strict licence to be included to OpenCV probably it could be of some help to your research.

[sudoankit]

Yes, I have used it but unfortunately it doesn't do what we are trying to achieve. Thank you!

[themechanicalcoder]

@terfendail I am interested in this project if no one is working on this should I start working on this issue?

[terfendail]

As far as I can suppose from current trends in detector development it is highly probable that this research will end up as DNN model that should be contribution for different repository. Anyway at the moment there is no active development related to this issue that I am aware of so you're welcome.