search

anibali / eval-mpii-pose

Evaluation code for the MPII human pose dataset
64 stars 15 forks source link

Question about predition mat generation #5

Open ahangchen opened 6 years ago

ahangchen commented 6 years ago

When we do Multi person Estimation and generate the prediction mat

% Set predicted x_pred, y_pred coordinates and prediction score for each body joint
pred(imgidx).annorect(ridx).annopoints.point(pidx).x = x_pred;
pred(imgidx).annorect(ridx).annopoints.point(pidx).y = y_pred;
pred(imgidx).annorect(ridx).annopoints.point(pidx).score = score;

Should ridx be in rectidxs_multi_test (extracted by getMultiPersonGroups(groups,RELEASE,false))? Or it's just a iterator over the people in an image according to my preditions result?

anibali commented 6 years ago

I have no idea about multi-person estimation, sorry. I've only ever attempted the single person task.

ahangchen commented 6 years ago

OK, same question in single person way: When we generate pred_keypoints_mpii.mat for single person evaluation, what's ridx referring to?

pred = annolist_test;
pred(imgidx).annorect(ridx).annopoints.point(pidx).x = x_pred;
pred(imgidx).annorect(ridx).annopoints.point(pidx).y = y_pred;
pred(imgidx).annorect(ridx).annopoints.point(pidx).id = id;

For example, for img 1, we predict 4 persons. But in the ground truth, single person field infers that img 1 only contains 2 single person. In this condition, what's the range of ridx when we prepare the mat? 1\~4 or 1\~2? Should we filter out some results or sort them according to the single person field?

ahangchen commented 6 years ago

In multi person evaluation, the problem may be more serious. If we predict a single person that doesn't belong to any group. It will be treated as false prediction, but actually, there is indeed one corresponding person in the image. This makes multi person evaluation weired.

anibali commented 6 years ago

Single person evaluation

Each example has a unique combination of imgidx and ridx. In the .h5 annotation files (which are derived from the original .m annotations), all of the examples are flattened, and can be indexed linearly by an example index. You can look up the original ridx and imgidx values for a specific example from the .h5 by using the example index to look up values from /person and /index respectively. Here's an example of looking up ridx from my code for preparing test results:

https://github.com/anibali/eval-mpii-pose/blob/2ae99137b02b2d1b3d221e8a042ecfc5206b424b/prepareTestResults.m#L79

In this instance, i is the example index, idx is the value of imgidx for the example, and test_persons is used to look up the ridx value for the example.

So if there are only examples for two people in image 1, that's no problem---just look up what their ridx values are from the annotations, and fill in the joint predictions appropriately.

Multi person evaluation

I agree that the multi person evaluation protocol looks confusing. Like I said, I have not worked with it myself. You should probably contact the actual creators of the dataset to try and get things clarified. If you do work it out, please post your findings here to help other people in the future.