Export: Problems with track_ids and empty objects

[dari1495]

sequence 015933, frame 0, track id 3. After export track id is 5

/home/cvg_anno/Datasets/posetrack_data_export/val/015933_mpii_test.json

{"category_id": 1, "track_id": 0, "person_id": 32351, "bbox_head": [603.869739478958, 386, 693.869739478958, 510], "id": 1015933000001, "image_id": 10159330000, "scores": [], "bbox": [], "keypoints": [643, 463, 1, 644, 514, 1, 649, 414, 1, 0, 0, 0, 0, 0, 0, 705, 553, 1, 563, 525, 1, 723, 680, 1, 519, 620, 1, 710, 695, 1, 534, 696, 1, 660, 703, 1, 590, 713, 1, 682, 904, 1, 606, 925, 1, 675, 1037, 1, 621, 1036, 1]},

{"category_id": 1, "track_id": 1, "person_id": 32351, "bbox": [1230.1128880526812, 427.1186440677966, 279.96237064910633, 656.9491525423728], "id": 1015933000001, "image_id": 10159330000, "scores": [], "bbox_head": [], "keypoints": []}

From the same file. Track ids 0 and 1 have the same person id

[dari1495]

What I think is happening:

• In certain videos, swapping of person and bbox_head's track_id leads to unordered lists in the database.
• Unordered lists in the database leads to problems in the export script with the track_id lookup
• These problems lead to objects missing in the json files
• And this makes the person_id, that should always be that of the bbox, not be well defined in annotations where the bbox object is incorrectly missing

Summary: Check the way objects are added into the file because it's leading to missing objects

[dari1495]

As far as I found there are several scenarios where there are more than one bboxes with the same person_id

1) There is a change in the camera view and the bboxes change track_id, this causes two different bbox to have the same person_id and different track_id, though there is always only one annotated at the same time. 2) There are two bboxes with the same person_id, but only one is annotated and the other one is empty. This is not a problem, but, this causes that bbox_head and person are also duplicated and sometimes the corresponding bbox_head and person objects are annotated on the track_id belonging to the bbox that is empty. This is worse because it creates two objects and while only one bbox is annotated at the same time, the correspondence with bbox_head and person is lost. 3) There is an actual error in annotation and there are two persons with the same person_id and different track_id. I can't do nothing about this. 4) The person is actually present twice in the same scene (004891 first 4 frames)

Now, what I propose that would fix 1) and 2) is changing the way I export the data. Instead of processing objects as they come, I would: I) Separate them into 3 arrays, one for bbox, one for bbox_head and one for person II) Process bboxes first, merging them by person_id and keeping all the track_ids merged in an array II a) If there is only one bbox annotated at the same frame as per 1) and 2), this would keep the annotated keypoints and discard the empty array. If both are empty, it would remain empty. II b) If there is a case 3) I need you to tell me what to do. III) After that, process bbox_head and person, looking up its correspondences by track_id. Same problem applies, there should only be one annotated per frame. If there are more than one, I could: III a) Keep one and discard the rest III b) Create a different object, but this object would not have a reliable person_id.

If I'm not mistaken, this would solve 1) and 2) and other cases like that if they should arise. In the end, it would mean that in the exported file the person_id would be unique per frame, which is what you wanted. However, the issue of data integrity arises in cases where the videos are wrongfully annotated.