search

leonid-pishchulin / poseval

Evaluation of multi-person pose estimation and tracking
Other
211 stars 49 forks source link

The total MOTA does not equal to the 'average joint-level MOTA' #11

Closed jin-s13 closed 6 years ago

jin-s13 commented 6 years ago

The total MOTA does not equal to the 'average joint-level MOTA'

I added the following lines in eval_helpers.py :: def getCum(vals) to compute the average joint-level MOTA.

cum += [(vals[[Joint().right_ankle, Joint().right_knee, Joint().right_hip, Joint().left_hip, Joint().left_knee,
                   Joint().left_ankle, Joint().right_wrist, \
                   Joint().right_elbow, Joint().right_shoulder, Joint().left_shoulder, Joint().left_elbow,
                   Joint().left_wrist, Joint().neck, Joint().nose, Joint().head_top], 0].mean())]

I find that it does not equal to the total MOTA ! The total MOTA is much lower (about 5%).

hirotomusiker commented 6 years ago

I have also found that : in the 'computeMetrics' function of evaluateAP.py, apAll is firstly defined as zeros with length of (nGTall.shape[0] + 1), apAll[0:nGTall.shape[0]] are replaced with ap values, and finally the joint average is calculated including the last zero:

    idxs = np.argwhere(~np.isnan(apAll[:,0]))
    apAll[nGTall.shape[0]] = apAll[idxs, 0].mean()

Is this how it's supposed to be?

leonid-pishchulin commented 6 years ago

@hirotomusiker: indeed this is a bug. I fixed this bug and pushed into the repo, see https://github.com/leonid-pishchulin/poseval/commit/0268b263c82a61fd4758826746f4012cdf22da71