MOTA revisited

[nikk-nikaznan]

I am trying to add more definition to what we consider as ID switches. This PR consider ID switch as a crab been assigned a new ID (re-ID)

• This is a bug : The old way of getting the ID switch is by finding the symmetric_difference between two list. The problem with this if we have prev_id = [1, 2, 3] and current_id = [1, 2, 4] the n_switches =2. The new approach will be 1.

After some help from @sfmig and @samcunliffe, this is currently what we have:

• We create a dict to map the ground_truth ID to tracked ID in every frame.

  • during count_identity_switches we have prev_frame_id_map and current_frame_id_map.
  • for each gt ID and tracked ID of current frame, we retrieve the tracked ID from previous frame that associated with a particular ground truth ID in current frame.
  • we also do the inverse, we retrieve the ground truth ID from previous frame that associate with a particular predicted ID in current frame.
    • we only check once, if the previous ID either from ground truth or prediction exist, if they exist, we check if the value is the same.

• Some changes for the test -- feel free to play around with the value in the test to see any other possibility.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 72.72727% with 33 lines in your changes missing coverage. Please review.

Project coverage is 46.16%. Comparing base (ff59d84) to head (1d45dd6).

Files	Patch %	Lines
crabs/tracker/evaluate_tracker.py	80.43%	18 Missing :warning:
crabs/tracker/track_video.py	40.00%	15 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #181 +/- ## ========================================== + Coverage 39.52% 46.16% +6.64% ========================================== Files 24 24 Lines 1417 1460 +43 ========================================== + Hits 560 674 +114 + Misses 857 786 -71 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[sfmig]

hey @nikk-nikaznan, I was just having a look at this and I get a different result:

Similarly, if prev_id = [1, 2, 3] and current_id = [1, 2, 3, 4] with current_gt = 4, n_switches = 0. (A new ID is appear in the current frame)

For me this returns n_switches=1

I think this is not intended right? Otherwise whenever a crab appears in view (exiting a burrow for example) it will be counted as an ID switch.

[sfmig]

I think there is some confusion, because as I understand it from the HOTA paper we only check ID switches for the true positives. So we shouldn't be filtering out missed detections or false negatives inside this function, right?

I got this from sections 3 and 4.1 (included). I recommend it, it is very well explained.

I think we could rewrite the count_identity_switches function to count ID switches as they say in the paper. Following their approach, we could fix the error above and also detect if IDs are swapped between crabs (right now, we don't detect this if all IDs are present in frame f and frame f-1).

[sfmig]

I think it would be something like follows:

1. For the bounding boxes that match the ground truth in frame f (i.e., that overlap sufficiently with the ground truth box), compute the map (dictionary) from ground truth IDs to predicted IDs.
2. Get the same map for the previous frame f-1 (we would have stored it).
3. The number of ID switches is the number of times the following thing occurs: for every ground-truth ID (keys of the dictionary), when do the predicted IDs disagree?

Let me know thoughts, happy to chat offline if you prefer.

[samcunliffe]

@sfmig I think there is some confusion, because as I understand it from the HOTA paper we only check ID switches for the true positives. So we shouldn't be filtering out missed detections or false negatives inside this function, right?

https://github.com/JonathonLuiten/TrackEval/blob/12c8791b303e0a0b50f753af204249e622d0281a/trackeval/metrics/clear.py#L93-L97

[nikk-nikaznan]

Thanks Sofia!

It would be nice that if we pass the ground truth and request an output video, both detections and ground truth are plotted in the video. This may be too big for this PR tho, so no worries if it's a faff.

I do have some code somewhere, because I did show them last TI. I did in a rush, + jet lagged. I will open an issue on that, should be not that hard, but a bit messy so will not add in this PR

I think I have address most of your comment. We can go through the changes tomorrow during PP.