search

thuml / Transfer-Learning-Library

Transfer Learning Library for Domain Adaptation, Task Adaptation, and Domain Generalization
http://transfer.thuml.ai
MIT License
3.39k stars 553 forks source link

Question about the PCK threshold setting about the evaluation of keypoint detection #178

Closed delaprada closed 1 year ago

delaprada commented 1 year ago

Hello!

I read the paper of "Regressive Domain Adaptation for Unsupervised Keypoint Detection" today and found that the PCK threshold setting is described in the paper as below:

An estimation is considered correct if its distance from the ground truth is less than a fraction α = 0.05 of the image size.

But when I read the source code: https://github.com/thuml/Transfer-Learning-Library/blob/b731e85e8e9eb79315da54a6fbe3c3efbf810adf/examples/domain_adaptation/keypoint_detection/erm.py#L247 I found that the accuracy function strictly set the threshold to 0.5.

So which one should I choose when I do my experiment would be better? A fraction α = 0.05 of the image size or 0.5?

Thank you.

JunguangJiang commented 1 year ago

The evaluation code is adapted from https://github.com/microsoft/human-pose-estimation.pytorch/blob/18f1d0fa5b5db7fe08de640610f3fdbdbed8fb2f/lib/core/evaluate.py.

The accuracy function strictly set the threshold to 0.5, yet it corresponds to a fraction α = 0.05 of the image size. See code https://github.com/thuml/Transfer-Learning-Library/blob/b731e85e8e9eb79315da54a6fbe3c3efbf810adf/tllib/utils/metric/keypoint_detection.py#L79

        norm = np.ones((pred.shape[0], 2)) * np.array([h, w]) / 10
delaprada commented 1 year ago

Thank you for your kindly explanation! I understand what you mean. @JunguangJiang