Use skeleton to extract pose with pose task

[darouwan]

Search before asking

• [X] I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

I am trying to use pose model to detect a person's pose and extract the position of arm, face, leg. The body position seems highly relate to the variable skeleton here: https://github.com/ultralytics/ultralytics/blob/e285d3d1b285ba80fa77d2a55a8875d477c8a826/ultralytics/utils/plotting.py#L138-L159

But I cannot understand how the self.skeleton is used here: https://github.com/ultralytics/ultralytics/blob/e285d3d1b285ba80fa77d2a55a8875d477c8a826/ultralytics/utils/plotting.py#L405-L418

Can you help to give some introduction on this?

Additional

No response

[glenn-jocher]

@darouwan hello,

Thank you for reaching out and for your thorough search before posting your question! 😊

To detect a person's pose and extract specific body parts like the arm, face, and leg using the YOLOv8 pose model, you are correct that the skeleton variable plays a crucial role. The skeleton essentially defines the connections between keypoints, which represent different parts of the body.

Here's a brief explanation of how the self.skeleton is used in the code:

1. Definition of Skeleton: The skeleton variable is a list of tuples, where each tuple represents a pair of keypoints that should be connected. For example, a tuple (5, 6) might represent a connection between the left shoulder and the left elbow.

   self.skeleton = [(5, 6), (6, 7), (7, 8), ...]  # Example skeleton connections

2. Usage in Plotting: In the plotting function, the self.skeleton is used to draw lines between the keypoints. This visually represents the pose by connecting the dots (keypoints) according to the defined skeleton.

   for i, j in self.skeleton:
      if i in keypoints and j in keypoints:
          cv2.line(img, keypoints[i], keypoints[j], color, thickness)

   Here, keypoints is a dictionary where the keys are the indices of the keypoints, and the values are their coordinates. The loop iterates over each pair in the self.skeleton and draws a line between the corresponding keypoints if they exist.

Example Code

To help you get started, here's a simplified example of how you might use the pose model to detect and plot keypoints:

from ultralytics import YOLO
import cv2

# Load the pose model
model = YOLO("yolov8n-pose.pt")

# Run inference on an image
results = model("path/to/image.jpg")

# Extract keypoints
keypoints = results[0].keypoints.xy

# Define skeleton connections (example)
skeleton = [(5, 6), (6, 7), (7, 8)]  # Define your skeleton connections here

# Plot keypoints and skeleton
img = cv2.imread("path/to/image.jpg")
for i, (x, y) in enumerate(keypoints):
    cv2.circle(img, (int(x), int(y)), 5, (0, 255, 0), -1)  # Draw keypoints

for i, j in skeleton:
    if i < len(keypoints) and j < len(keypoints):
        cv2.line(img, (int(keypoints[i][0]), int(keypoints[i][1])), 
                      (int(keypoints[j][0]), int(keypoints[j][1])), (255, 0, 0), 2)  # Draw skeleton

cv2.imshow("Pose", img)
cv2.waitKey(0)
cv2.destroyAllWindows()

This example demonstrates how to load a pose model, run inference, extract keypoints, and plot them along with the skeleton connections.

For more detailed information on pose estimation and additional resources, you can visit our Pose Estimation Documentation.

I hope this helps! If you have any further questions, feel free to ask.

[darouwan]

@glenn-jocher So the indeces of keypoints match to specific body joint, right? And there is 17 keypoints in default pose model , can I know what each index means here?

[glenn-jocher]

Hello @darouwan,

Yes, you are correct! The indices of the keypoints correspond to specific body joints in the pose estimation model. In the default YOLOv8 pose model, there are 17 keypoints, each representing a different part of the body. Here is the mapping of each index to its respective body joint:

1. 0: Nose
2. 1: Left Eye
3. 2: Right Eye
4. 3: Left Ear
5. 4: Right Ear
6. 5: Left Shoulder
7. 6: Right Shoulder
8. 7: Left Elbow
9. 8: Right Elbow
10. 9: Left Wrist
11. 10: Right Wrist
12. 11: Left Hip
13. 12: Right Hip
14. 13: Left Knee
15. 14: Right Knee
16. 15: Left Ankle
17. 16: Right Ankle

This mapping allows you to identify and work with specific body parts based on their keypoint indices.

If you are interested in visualizing these keypoints, you can use the example code provided earlier to draw the keypoints and their connections on an image. This can help you better understand how the keypoints correspond to different body parts.

For more detailed information on pose estimation and additional resources, you can visit our Pose Estimation Documentation.

If you have any further questions or need additional assistance, feel free to ask. We're here to help! 😊

[darouwan]

@glenn-jocher thanks! That is what need currently and it's very useful for pose model. Do you mind to add it as comment in code and explain it in documents? I am happy to refine it .

[glenn-jocher]

Hello @darouwan,

Thank you for your positive feedback! 😊 We're glad to hear that the information provided was useful for your work with the pose model.

Adding detailed comments in the code and expanding the documentation to include explanations of keypoint indices and their corresponding body parts is a great suggestion. This would certainly help other users better understand and utilize the pose estimation features.

To contribute to the documentation and code comments, you can follow these steps:

1. Fork the Repository: Start by forking the Ultralytics repository to your GitHub account.
2. Make Changes: Add the necessary comments in the code and update the documentation with the explanations.
3. Submit a Pull Request: Once you've made the changes, submit a pull request (PR) to the main repository. Our team will review your PR and merge it if everything looks good.

For detailed instructions on contributing, you can refer to our Contributing Guide.

Here's a quick example of how you might add comments to the code:

# Keypoint indices for pose estimation
# 0: Nose, 1: Left Eye, 2: Right Eye, 3: Left Ear, 4: Right Ear
# 5: Left Shoulder, 6: Right Shoulder, 7: Left Elbow, 8: Right Elbow
# 9: Left Wrist, 10: Right Wrist, 11: Left Hip, 12: Right Hip
# 13: Left Knee, 14: Right Knee, 15: Left Ankle, 16: Right Ankle

And for the documentation, you might add a section explaining the keypoint indices and their corresponding body parts.

We appreciate your willingness to refine and improve the documentation. Your contributions are valuable to the community!

If you have any questions or need further assistance with the contribution process, feel free to ask. We're here to help!

[darouwan]

Hi @glenn-jocher I have submit a new PR to refine it.

[glenn-jocher]

Hello @darouwan,

Thank you for your contribution! 🎉 We appreciate your effort in refining the documentation and code comments for the pose model. Your PR will be reviewed by our team shortly.

If you have any further suggestions or improvements, feel free to share them. Your input helps make the Ultralytics community even better!