search

cheind / py-motmetrics

:bar_chart: Benchmark multiple object trackers (MOT) in Python
MIT License
1.37k stars 259 forks source link

What's the definition of num_transfer, num_ascend, num_migrate? #104

Closed shen338 closed 4 years ago

shen338 commented 4 years ago

First of all, thank you for the amazing work! Very useful.

I have some confusions about the definition of num_transfer, num_ascend, num_migrate. From previous issues, I found these:

I define two extra matching states: TRANSFER and ASCEND. A TRANSFER means a hypothesis ID is assigned from an old target to a new target (the SWITCH means a target is assigned from an old hypothesis ID to a new hypothesis ID). An ASCEND means a new ID is assigned to a target (this is a sub-state for SWITCH). For instance, a->b, b->a is TRANSFER also SWITCH(not ASCEND), a->b(new) is TRANSFER but not SWITCH, c(new)->a is SWITCH also ASCEND, but not TRANSFER. In the origin metrics, there exists mistake cases cannot be modeled by SWITCH, but can be figured out by TRANSFER. The purpose is to make more detailed analysis.

Based on this, I kind of understand the meaning of TRANSFER, but not so sure about ASCEND and MIGRATE.

And also, I found this information, making me more confused:

# assert ((summary['num_transfer'] - summary['num_migrate']) == (summary['num_switches'] - summary['num_ascend'])).all() # False assertion

Could you kindly explain this with some examples? Thank you!

@Helicopt @cheind

andreilica commented 4 years ago

I'm also curious about this. Also, lower numbers in those 3 metrics indicate better performance? Or how they should be interpreted?

jvlmdr commented 4 years ago

The MOT16 paper defined "identity switches" (num_switches) to be the number of times that the predicted identity of a ground-truth track changed. However, the converse is not counted, i.e. the number of times that the ground-truth identity of a predicted track changes.

Consider the following example. Track 1 is following object A until it leaves the scene. At the same time, object B enters the scene and track 1 jumps ("transfers") from object A to object B. This would not count as a "SWITCH" event and would not affect num_switches, since the predicted identities of object A and object B did not change.

If I understand correctly, then the "TRANSFER" event and the num_transfer statistic were created to count these types of errors.

jvlmdr commented 4 years ago

I believe these events are most clearly defined in the docstring of MOTAccumulator. Compare the definitions of MATCH, SWITCH, TRANSFER, ASCEND, MIGRATE.

https://github.com/cheind/py-motmetrics/blob/bdc417d92e4c2e15ba2a6e22579f78c2e78c3334/motmetrics/mot.py#L43-L50

andreilica commented 4 years ago

Ok, so all of SWITCH, TRANSFER, ASCEND and MIGRATE should be as low as possible for a correct person reID?

On Mon, 1 Jun 2020 at 23:26, Jack Valmadre notifications@github.com wrote:

I believe these events are most clearly defined in the docstring of MOTAccumulator. Compare the definitions of MATCH, SWITCH, TRANSFER, ASCEND, MIGRATE.

https://github.com/cheind/py-motmetrics/blob/bdc417d92e4c2e15ba2a6e22579f78c2e78c3334/motmetrics/mot.py#L43-L50

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cheind/py-motmetrics/issues/104#issuecomment-637084005, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPOGX6TAPMEGKAEDCRUZR3RUQFIFANCNFSM4NID24HQ .

jvlmdr commented 4 years ago

Yep, lower is better for all.

cheind commented 4 years ago

I close this problem as solved. If not, feel free to reopen.