MOTA calculation query

[bml1g12]

I am using motmetrics==1.2.0

Evaluating sequence: One-Person-Left-Right-Video from SimEasy in group Simulation:
      IDF1    IDP   IDR  Rcll   Prcn GT MT PT ML FP FN IDs  FM  MOTA  MOTP IDt IDa IDm
full 96.3% 100.0% 92.9% 92.9% 100.0%  1  1  0  0  0  1   0   0 92.9% 0.000   0   0   0
part 96.3% 100.0% 92.9% 92.9% 100.0%  1  1  0  0  0  1   0   0 92.9% 0.000   0   0   0
> /data/gym_ai/TrackerEvaluation/evaluator.py(201)sim_evaluation()
-> mota = evaluation.mota.values[1]  # get the partial result
(Pdb) evaluation
          idf1  idp       idr    recall  precision  num_unique_objects  mostly_tracked  partially_tracked  ...  num_misses  num_switches  num_fragmentations      mota  motp  num_transfer  num_ascend  num_migrate
full  0.962963  1.0  0.928571  0.928571        1.0                   1               1                  0  ...           1             0                   0  0.928571   0.0             0           0            0
part  0.962963  1.0  0.928571  0.928571        1.0                   1               1                  0  ...           1             0                   0  0.928571   0.0             0           0            0

[2 rows x 18 columns]
(Pdb) 1 - (evaluation.num_misses + evaluation.num_false_positives + evaluation.num_switches) / evaluation.num_unique_objects

When I use PDB debugger to perform this equation from the MOTA code https://github.com/cheind/py-motmetrics/blob/d261d16cca263125b135571231011ccf9efd082b/motmetrics/metrics.py#L515 I get a MOTA of 0:

full    0.0
part    0.0
dtype: float64

How is MOTA calculated in this library to obtain 92.9% from this data? Struggling to fine the code that is actually called to do this calculation; as I guess it can't be that which I linked.

Kind Regards,

[cheind]

Hey, would you mind providing a reproducable MWE? Seems like your dataset is small enough, that this should be quite easy.

[cheind]

As far as code is concerned, there are ongoing efforts to refactor it #118

[bml1g12]

Hello, thank you for your quick response. I have attached a minimum working example. It is slightly different to above but the same idea; trying to understand what calculation is performed to compute MOTA.

([], []) (array([0]), array([[31, 85, 38, 67]]))
(['0'], [array([35, 86, 42, 65])]) (array([0]), array([[35, 86, 42, 65]]))
(['0'], [array([40, 87, 45, 64])]) (array([0]), array([[40, 87, 45, 64]]))
(['0'], [array([47, 87, 44, 64])]) (array([0]), array([[47, 87, 44, 64]]))
(['0'], [array([53, 87, 40, 64])]) (array([0]), array([[53, 87, 40, 64]]))
(['0'], [array([58, 86, 36, 65])]) (array([0]), array([[58, 86, 36, 65]]))
(['0'], [array([63, 86, 33, 66])]) (array([0]), array([[63, 86, 33, 66]]))
(['0'], [array([67, 85, 34, 66])]) (array([0]), array([[67, 85, 34, 66]]))
(['0'], [array([73, 86, 38, 65])]) (array([0]), array([[73, 86, 38, 65]]))
(['0'], [array([78, 86, 37, 65])]) (array([0]), array([[78, 86, 37, 65]]))
(['0'], [array([84, 87, 33, 65])]) (array([0]), array([[84, 87, 33, 65]]))
(['0'], [array([90, 87, 28, 65])]) (array([0]), array([[90, 87, 28, 65]]))
(['0'], [array([97, 87, 26, 65])]) (array([0]), array([[97, 87, 26, 65]]))
(['0'], [array([103,  87,  30,  64])]) (array([0]), array([[103,  87,  30,  64]]))
(['0'], [array([108,  87,  33,  64])]) (array([0]), array([[108,  87,  33,  64]]))
(['0'], [array([115,  88,  30,  64])]) (array([0]), array([[115,  88,  30,  64]]))
(['0'], [array([120,  87,  26,  64])]) (array([0]), array([[120,  87,  26,  64]]))
(['0'], [array([124,  86,  23,  65])]) (array([0]), array([[124,  86,  23,  65]]))
(['0'], [array([128,  86,  22,  66])]) (array([0]), array([[128,  86,  22,  66]]))
(['0'], [array([133,  86,  27,  66])]) (array([0]), array([[133,  86,  27,  66]]))
(['0'], [array([137,  86,  31,  64])]) (array([0]), array([[137,  86,  31,  64]]))
(['0'], [array([143,  87,  28,  65])]) (array([0]), array([[143,  87,  28,  65]]))
(['0'], [array([150,  87,  22,  65])]) (array([0]), array([[150,  87,  22,  65]]))
(['0'], [array([156,  87,  19,  65])]) (array([0]), array([[156,  87,  19,  65]]))
(['0'], [array([163,  87,  18,  65])]) (array([0]), array([[163,  87,  18,  65]]))
(['0'], [array([166,  87,  23,  64])]) (array([0]), array([[166,  87,  23,  64]]))
(['0'], [array([168,  88,  28,  64])]) (array([0]), array([[168,  88,  28,  64]]))
(['0'], [array([171,  88,  27,  64])]) (array([0]), array([[171,  88,  27,  64]]))
(['0'], [array([177,  87,  24,  64])]) (array([0]), array([[177,  87,  24,  64]]))
(['0'], [array([186,  86,  21,  66])]) (array([0]), array([[186,  86,  21,  66]]))
(['0'], [array([192,  86,  21,  67])]) (array([0]), array([[192,  86,  21,  67]]))
(['0'], [array([194,  86,  25,  65])]) (array([0]), array([[194,  86,  25,  65]]))
(['0'], [array([196,  87,  29,  65])]) (array([0]), array([[196,  87,  29,  65]]))
(['0'], [array([199,  87,  31,  65])]) (array([0]), array([[199,  87,  31,  65]]))
(['0'], [array([206,  87,  29,  65])]) (array([0]), array([[206,  87,  29,  65]]))
(['0'], [array([215,  87,  26,  65])]) (array([0]), array([[215,  87,  26,  65]]))
(['0'], [array([220,  87,  28,  65])]) (array([0]), array([[220,  87,  28,  65]]))
(['0'], [array([221,  87,  32,  64])]) (array([0]), array([[221,  87,  32,  64]]))
(['0'], [array([223,  88,  36,  64])]) (array([0]), array([[223,  88,  36,  64]]))
(['0'], [array([228,  87,  37,  64])]) (array([0]), array([[228,  87,  37,  64]]))
(['0'], [array([235,  86,  35,  65])]) (array([0]), array([[235,  86,  35,  65]]))
(['0'], [array([245,  86,  32,  66])]) (array([0]), array([[245,  86,  32,  66]]))
(['0'], [array([247,  86,  36,  66])]) (array([0]), array([[247,  86,  36,  66]]))
(['0'], [array([248,  86,  40,  65])]) (array([0]), array([[248,  86,  40,  65]]))
(['0'], [array([250,  86,  43,  65])]) (array([0]), array([[250,  86,  43,  65]]))
(['0'], [array([255,  87,  43,  65])]) (array([0]), array([[255,  87,  43,  65]]))
(['0'], [array([263,  86,  41,  65])]) (array([0]), array([[263,  86,  41,  65]]))
(['0'], [array([271,  86,  39,  65])]) (array([0]), array([[271,  86,  39,  65]]))
(['0'], [array([274,  86,  42,  65])]) (array([0]), array([[274,  86,  42,  65]]))
(['0'], [array([275,  87,  45,  65])]) (array([0]), array([[275,  87,  45,  65]]))
(['0'], [array([277,  87,  47,  65])]) (array([0]), array([[277,  87,  47,  65]]))
(['0'], [array([282,  87,  47,  65])]) (array([0]), array([[282,  87,  47,  65]]))
      IDF1    IDP   IDR  Rcll   Prcn GT MT PT ML FP FN IDs  FM  MOTA  MOTP IDt IDa IDm
full 99.0% 100.0% 98.1% 98.1% 100.0%  1  1  0  0  0  1   0   0 98.1% 0.000   0   0   0
part 99.0% 100.0% 98.1% 98.1% 100.0%  1  1  0  0  0  1   0   0 98.1% 0.000   0   0   0
Mota using equation is:
full    0.0
part    0.0
dtype: float64
but MOTA reported is  0.9807692307692307

Attached is the code here:

MinimumWorkingExample.zip

p.s. The motivation for this is I have a pathological example:

TrackerPathologicalExample.zip

With lots of false positives in the tracking; and wanted to make sure the negative -2641.7% MOTA was expected behaviour - which I expect it is in such a case as we have so many false positives.

[cheind]

Hey!

Thanks for the example. What's the output ([], []) (array([0]), array([[31, 85, 38, 67]])) (['0'], [array([35, 86, 42, 65])]) (array([0]), array([[35, 86, 42, 65]]))...?

[bml1g12]

Thanks,

this is print(tracker_data, gt_data) - for tracker and ground truth data respectively.

each of these is themselves composed of a tuple of (ids, boxes)

Where boxes is a list of bounding boxes (x,y,w,h) and ids is a list if integer ids, starting from 0

You can see the script for details

[cheind]

How is MOTA calculated in this library to obtain 92.9% from this data?

MISS 1 (frame 0) NUMOBJECTS 52 It uses num objects not num unique objects. -> 1 - (1/52) = 0.98

See section 2.2 of

https://cvhci.anthropomatik.kit.edu/~stiefel/papers/ECCV2006WorkshopCameraReady.pdf

it divides by sum(gt) and states Let also gt be the number of objects present at time t.

[bml1g12]

Ah I understand now; I need to calculate the sum(ground truth objects) for the denominator. I confirmed in my pathological example I get 1-(329/12) resulting in the huge negative MOTA so I think all is good here and we can close this ticket - thanks.