ultralytics / ultralytics

Ultralytics YOLO11 🚀
https://docs.ultralytics.com
GNU Affero General Public License v3.0
36.58k stars 7.05k forks source link

Why is there no conf threshold for keypoints? #18960

Open alegkov opened 2 weeks ago

alegkov commented 2 weeks ago

Search before asking

Question

I found the conf = 0.5 in this line https://github.com/ultralytics/ultralytics/blob/df6572a58b19ea1e79d5d7747bb151d4509c2cc6/ultralytics/engine/results.py#L1385, and the behavior was unexpected for me, because the coordinates are zeroed, why not return the original value, and add the conf to the postprocessing for filtering keypoints?

Additional

No response

UltralyticsAssistant commented 2 weeks ago

👋 Hello @alegkov, thank you for bringing up this question 🚀! We recommend checking out the Docs, where you can learn more about keypoints and YOLO's postprocessing workflows. You might also find useful insights in the Python and CLI usage examples.

Regarding the behavior you mentioned, if you believe it might be a bug 🐛, it would be helpful if you could provide a minimum reproducible example to help us better understand and investigate the issue.

For additional clarifications or discussions, feel free to connect with the Ultralytics community:

Upgrade

If you're using an older version of the ultralytics package, please make sure you upgrade to the latest version along with all requirements in a Python>=3.8 environment. This ensures you're testing with the latest updates:

pip install -U ultralytics

Environments

If you'd like to test or reproduce the functionality in a fresh environment, you can use the following verified setups:

Status

Ultralytics CI

When this badge is green, all Ultralytics CI tests are successfully passing, confirming compatibility and correctness across multiple operating systems.

📢 This is an automated response to get you started, but rest assured, an Ultralytics engineer will review this and assist you shortly!

glenn-jocher commented 2 weeks ago

@alegkov thank you for your question! The confidence threshold of 0.5 in the keypoints processing is a design choice to automatically filter low-confidence points while preserving the data structure. The original confidence values remain accessible via the keypoints.conf property (Keypoints docs), allowing you to implement custom postprocessing filters as needed. This implementation balances usability with flexibility for different applications.

alegkov commented 1 week ago

@glenn-jocher thank you for your answer, ok i can get conf, but how can i get original xy?

glenn-jocher commented 1 week ago

@alegkov you can access the original unprocessed keypoint coordinates (before confidence thresholding) through the keypoints.data attribute. For example:

original_xy = keypoints.data[..., :2]  # Shape [N, K, 2]

This gives the raw coordinates from the model output before any confidence-based masking is applied. For more details, see the Keypoints class documentation.

alegkov commented 1 week ago

I checked and I also get 0.0 for keypoint coordinates when I use: original_xy = keypoints.data[..., :2] # Shape [N, K, 2]

the thing is that the data is essentially

mask = kpts[..., 2] 
kpts[..., :2][mask] = 0

keypoints.data = kpts

Results

class Results
    def __init__(self, keypoints, orig_shape) -> None:
        ...
       self.keypoints = Keypoints(keypoints, self.orig_shape) if keypoints is not None else None
       ...

Keypoints

class Keypoints(BaseTensor):
    def __init__(self, keypoints, orig_shape) -> None:
        ...
        if keypoints.shape[2] == 3:  # x, y, conf
            mask = keypoints[..., 2] < 0.5  # points with conf < 0.5 (not visible)
            keypoints[..., :2][mask] = 0
        super().__init__(keypoints, orig_shape)

BaseTensor

class BaseTensor(SimpleClass):
    def __init__(self, data, orig_shape) -> None:
        ...
        self.data = data 
        ...
glenn-jocher commented 1 week ago

@alegkov you'd have to modify or remove this thresholding in a local branch to access the underlying values.