search

davidpagnon / Sports2D

Compute 2D human pose and angles from a video or a webcam.
BSD 3-Clause "New" or "Revised" License
46 stars 6 forks source link

Face + hands #2

Open p0mad opened 1 year ago

p0mad commented 1 year ago

Hi, your work is much appritiated, but do you plan to add hands + facial keypoints?

or at least guide on how to do it?

Thanks Best regards

davidpagnon commented 1 year ago

Thank you! I could definitely add these keypoints, although the consistency of results will highly depend on the image quality.

However, I am not sure which hand an face angles you'd be interested in? Or do you just need keypoint positions?

p0mad commented 1 year ago

@davidpagnon Thanks for the prompt reply! I'm on ubuntu 20.04 and first i want to extract keypoint and pose of the human in the video! I'm not intrested in angles of face, but i need the full capable hand and pose keypoint detection! ( not sure if the 3D is better or 2D is more accurate!) what do you recommand? are there more faster/accurate models than openpose?

I can also work with windows but ubuntu is preferred!

davidpagnon commented 1 year ago

If you just want positions, it's definitely doable and I can add this functionality next week. I would use the BODY_135 model (page here and article there) . It is generally acknowledged that OpenPose is one of the best compromises between accuracy / speed / ease of use, although AlphaPose and Blazepose have their strengths in slightly different areas.

You could run Sports2D on Ubuntu, or run it much more easily and regardless of your platform with the Colab version.

Now if you need 3D, this is a different issue. Some models infer it from a single video but they are not usually known to be very accurate yet. You can have research-grade results if you use several cameras, for example by using another open-source package I developed, Pose2Sim. Cameras need to be calibrated and synchronized, which is not trivial (although I'm currently developing a calibration procedure that may be easier to use, and that I hope to release in the next few weeks).

Pose2Sim is mostly used to obtain angles but can also give 3D positions. However, right now face keypoints are not triangulated. I could implement it but I would need a little more time.

p0mad commented 1 year ago

@davidpagnon Thank you so much for the guidance. I would definitely wait for your next week work! meantime, Would you please give some insight on how to use OpenPose on ubuntu?

I do get error on dependancies for my ubuntu 20.04! Would you please mention the path to install and use the demo of openpose? is it just cloning the repo and make install?

beside alpahpose and blazepose, i think mediapipe is also good, alongside the OSX https://osx-ubody.github.io/

Do you think they are also possible to be used?

davidpagnon commented 1 year ago

Well I have never installed it on Linux on a proper computer, but I did on Colab m, which works on Ubuntu 20.04.

You can access the code by double clicking on the "Installation" cell there http://bit.ly/Sports2D_Colab And then reproduce the steps in the function install_openpose_func().

BlazePose is actually the pose estimation solution of MediaPipe, by Google :)

PS: Sorry for closing and reopening the issue, my baby did this for me 😅

davidpagnon commented 1 year ago

And I haven't tried OSX yet, although it belongs to the SMPL suite. SMPL gives a 3D shape from 2d keypoints. Since there is a complete texture, it often looks quite good, but keypoints are not necessarily better estimated.

Could you tell me more specifically about your end-goal, so that I can understand better whether this package could be appropriate or not?

p0mad commented 1 year ago

@davidpagnon Great, I was checking your code few hours back and i found that specific cmake and specific cuda, cudnn version needs to be installed! am i right? or any other version is acceptable?

About the blazePose, i have doubt about the difference result with mediapipe holistic! ( it supports up to 543 keypoints! ( thats fascinating) but blazePose i think is the base model! is there anyway to use Mediapipe holistics all keypoints?

p0mad commented 1 year ago

@davidpagnon I want to use the KeyPoints or as you said the complete texture (like OSX) to use in stable diffusion as a guidance!

Is there a more human friendly environments we can talk together? ( like Telegram, discord, whats app or skype)

davidpagnon commented 1 year ago

Sorry,

I somehow missed your message!

@davidpagnon Great, I was checking your code few hours back and i found that specific cmake and specific cuda, cudnn version needs to be installed! am i right? or any other version is acceptable?

Yes indeed, I had to try a few versions and check for the experience of other users to find something that worked.

About the blazePose, i have doubt about the difference result with mediapipe holistic! ( it supports up to 543 keypoints! ( thats fascinating) but blazePose i think is the base model! is there anyway to use Mediapipe holistics all keypoints?

Actually, you are right! MediaPipe proposes a ton of solutions for different purposes, some of them focusing on keypoint estimation. BlazePose focuses on body pose, and Holistic includes blazepose and adds a lot of other hand and face ones.

davidpagnon commented 1 year ago

@davidpagnon I want to use the KeyPoints or as you said the complete texture (like OSX) to use in stable diffusion as a guidance!

Is there a more human friendly environments we can talk together? ( like Telegram, discord, whats app or skype)

It may be easier indeed, so that I can better frame your goals and how it would work. You can reach me with any media (Telegram, Signal, WhatsApp, phone, email, discord or anything). If you need my number, it can be found easily on the internet by typing my name :)