Memory usage

[headdab]

I'm running this with for a 900 frame sequence and it brings my system to its knees, consuming probably 20GB of memory. Any ideas?

[cheind]

900 frames shouldn't be a problem, unless you are using it in detection mode and computing global ID metrics -> large matrices. Global metrics are not really useful except in the tracking case, if you don't need them I'd advise to not compute them during mh.compute.

[headdab]

It wasn't using as much memory as I thought, although it is pretty slow.

Does metrics=mm.metrics.motchallenge_metrics compute these global metrics?

[cheind]

Yes it does [1]. When it is slow and you need the global metrics, you may want to try a different LAP solver. Try pip install lapsolver and see if you can gain some performance [2].

[1] https://github.com/cheind/py-motmetrics/blob/master/motmetrics/metrics.py#L416 [2] https://github.com/cheind/py-lapsolver

[headdab]

I ran it again and it indeed uses tons of memory, with a 900 frame video comparison. It actually still running and my machine is swapping.

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
15559 estes 20 0 26.0G 12.8G 1116 D 0.0 81.7 43:05.76 python ./detection-scores.py videos/gold/latest/0a.h5 0a-gauss.h5

I'll disable the global measurements for now, but MOT is all about tracking and I hope to try that soon. I'll also try the lapsolver.

[cheind]

Ok let me know how your test turns out. The global measurements are an addition to MOT metrics, so they are not a must but a nice to have feature.

[cheind]

Closing for now, reopen if needed.

[bbrudevold]

I am having a similar issue while trying to score the CVPR 2019 Tracking Challenge cases with "metrics=mm.metrics.motchallenge_metrics". It runs for hours while consuming nearly all my RAM (up to 20 GB). The same cases can be scored with the official Matlab scorer from the challenge in under a minute. I tried using lapsolver and lapjv and didn't see significant improvement. Any other suggestions? Thanks!

[cheind]

Could you give the develop branch a try and also disable metrics that are not required. Especially the global ID metrics.

Best, Chris

Bryan B notifications@github.com schrieb am Do. 9. Jän. 2020 um 21:12:

I am having a similar issue while trying to score the CVPR 2019 Tracking Challenge cases with "metrics=mm.metrics.motchallenge_metrics". It runs for hours while consuming nearly all my RAM (up to 20 GB). The same cases can be scored with the official Matlab scorer from the challenge in under a minute. I tried using lapsolver and lapjv and didn't see significant improvement. Any other suggestions? Thanks!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/cheind/py-motmetrics/issues/20?email_source=notifications&email_token=AAAJNJOSZRRPLCGJAU7BZFTQ46ALJA5CNFSM4GEE4OB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIRT3SA#issuecomment-572734920, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJNJOCCIW5GLY7LJWWXPDQ46ALJANCNFSM4GEE4OBQ .

[jvlmdr]

If you need the ID metrics, I have been working on an experimental change that uses a sparser representation for the global ID metrics. I haven't moved it to the official repo yet. You can find on the solvers branch of my forked repo https://github.com/jvlmdr/py-motmetrics/tree/solvers

Would you be able to upload your predictions so that I can take a look?

Thanks

[jvlmdr]

(It would be necessary to select a sparse solver using with set_default_solver(): either "ortools" or "lapmod")

[bbrudevold]

Thanks for the suggestions! I will try them out.

I have uploaded all my detection and groundtruth track files here. The smallest one (CVPR19-01) scores reasonably fast, but the others have difficulty finishing.

CVPR19-01.txt CVPR19-02.txt CVPR19-03.txt CVPR19-05.txt CVPR19-01_GT.txt CVPR19-02_GT.txt CVPR19-03_GT.txt CVPR19-05_GT.zip (zipped last one due to file size limit)

[bbrudevold]

I am running with metrics=mm.metrics.motchallenge_metrics and not disabling any metrics since I want the most information possible for tracker testing.

I did get some nice speedups from the alternate branches. These times are for the CVPR19-05 test case: Master branch with lapsolver: 100+ min and still not finished Develop branch with lapsolver: 23 min jvlmdr's experimental repo solvers branch with ortools solver: 32 min

The Matlab code took 13 seconds so there is still a large difference, but these are nice improvements.

[cheind]

Hi Bryan,

thanks for the update! The comparison of the runtime of py-motmetrics with the Matlab script you mentioned is only valid if we evaluate based on the equal metrics used. Especially global ID metrics take long to compute, since they cannot be calculated based individual frames. Which metrics are computed by the Matlab script? I quickly skimmed over https://arxiv.org/pdf/1906.04567.pdf (Table 6, and it appears that global ID metrics are not evaluated. Correct me if I'm wrong.

Best, Christoph

On Tue, Jan 14, 2020 at 10:51 PM Bryan B notifications@github.com wrote:

I am running with metrics=mm.metrics.motchallenge_metrics and not disabling any metrics since I want the most information possible for tracker testing.

I did get some nice speedups from the alternate branches. These times are for the CVPR19-05 test case: Master branch with lapsolver: 100+ min and still not finished Develop branch with lapsolver: 23 min jvlmdr's experimental repo solvers branch with ortools solver: 32 min

The Matlab code took 13 seconds so there is still a large difference, but these are nice improvements.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/cheind/py-motmetrics/issues/20?email_source=notifications&email_token=AAAJNJINDUUIVCL67P6WRADQ5YXVDA5CNFSM4GEE4OB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI6IE4A#issuecomment-574390896, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJNJP75A4QXV72ITTEQE3Q5YXVDANCNFSM4GEE4OBQ .

[jvlmdr]

Yeah, thanks a lot Bryan! These files are useful. The "CVPR19-05" sequence seems to dominate running time. It has ~300k boxes for ~3k frames (16MB txt file) and 4900 predicted identities for 1251 ground-truth identities. That's large but not unreasonable. It would be great if we can increase the speed.

So, I tried a few different configurations with your data (running apps.eval_motchallenge). Even without the id metrics, it does seem fairly slow (19 mins). All configurations take a huge amount of RAM too (around 40 or 50GB at peak). I'll investigate as soon as possible.

id metrics	solver	branch	mergeOverall (sec)	wall clock time	max resident (kbytes)
no	lapsolver	develop	377.804	19:48.14	47078776
no	lapsolver	solvers	357.517	19:40.36	47079948
no	lapmod	solvers	381.397	22:08.76	47083344
yes	lapsolver	develop	562.636	22:48.57	53678408
yes	lapsolver	solvers	529.655	22:22.82	53502648
yes	lapmod	solvers	552.205	24:57.14	53505376
greedy	lapsolver	solvers	528.749	22:47.58	53508264

It seems like adding the ID metrics does slow things down a bit, although this is even true when using the "greedy" versions of the ID metrics, which avoid solving a Linear Assignment Problem (LAP) for the identities (but still require the counting of all matches between predicted and ground-truth tracks).

The lapmod solver is slower than lapsolver. However, I think this is due to the per-frame LAP not the global identity LAP, since we see the same slow-down with the ID metrics disabled.

[cheind]

Thanks Jack for the in depth analysis!

On Wed, Jan 15, 2020 at 1:10 PM Jack Valmadre notifications@github.com wrote:

Yeah, thanks a lot Bryan! These files are useful. The "CVPR19-05" sequence seems to dominate running time. It has ~300k boxes for ~3k frames (16MB txt file) and 4900 predicted identities for 1251 ground-truth identities. That's large but not unreasonable. It would be great if we can increase the speed.

So, I tried a few different configurations with your data (running apps.eval_motchallenge). Even without the id metrics, it does seem fairly slow (19 mins). All configurations take a huge amount of RAM too (around 40 or 50GB at peak). I'll investigate as soon as possible. id metrics solver branch mergeOverall (sec) wall clock time max resident (kbytes) no lapsolver develop 377.804 19:48.14 47078776 no lapsolver solvers 357.517 19:40.36 47079948 no lapmod solvers 381.397 22:08.76 47083344 yes lapsolver develop 562.636 22:48.57 53678408 yes lapsolver solvers 529.655 22:22.82 53502648 yes lapmod solvers 552.205 24:57.14 53505376 greedy lapsolver solvers 528.749 22:47.58 53508264

It seems like adding the ID metrics does slow things down a bit, although this is even true when using the "greedy" versions of the ID metrics, which avoid solving a Linear Assignment Problem (LAP) for the identities (but still require the counting of all matches between predicted and ground-truth tracks).

The lapmod solver is slower than lapsolver. However, I think this is due to the per-frame LAP not the global identity LAP, since we see the same slow-down with the ID metrics disabled.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/cheind/py-motmetrics/issues/20?email_source=notifications&email_token=AAAJNJPL3NIMXXFBP7W4XR3Q534LXA5CNFSM4GEE4OB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJADFSQ#issuecomment-574632650, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJNJOBO6SBXAXPGZ57FETQ534LXANCNFSM4GEE4OBQ .

[bbrudevold]

Christoph,

I'm not sure which are considered to be global ID metrics, but the Matlab and py-motmetrics scorers appear to provide the same metrics other than the lack of FAR and MOTAL from py-motmetrics.

Here is the output from the Matlab scorer:

CVPR19-05 IDF1 IDP IDR| Rcll Prcn FAR| GT MT PT ML| FP FN IDs FM| MOTA MOTP MOTAL 36.1 55.2 26.8| 44.4 91.6 8.66| 1209 130 701 378| 28719394037 3426 12336| 39.9 88.0 40.4

And here is the py-motmetrics output:

       IDF1   IDP   IDR  Rcll  Prcn   GT  MT  PT  ML    FP     FN  IDs     FM  MOTA  MOTP

CVPR19-05 36.1% 55.2% 26.8% 44.4% 91.6% 1209 129 701 379 28717 394035 3387 12758 39.9% 0.120

[bbrudevold]

Jack, thanks for looking into this! I observed that around 70-80% of the time was spent adding frames to the accumulator, while computing metrics at the end was a smaller consumer. So it makes sense that disabling the ID metrics would only give a small speedup.

[cheind]

Bryan, yes the output looks identical! I was refering to the first three columns. In the paper they are not mentioned, so I assumed they are also not computed.

At least the numbers match (note: motp in motmetrics 1-motp(cvpr)). Event if 80 percent of the time is spent in adding frames, py-motmetrics still spends 2 mins in computation compared to a total of 13 seconds in matlab script.

[jvlmdr]

OK, I think I have found a way to get much better performance by reducing the number of RAW events. Currently, the Accumulator adds a RAW event for all (object, hypothesis) pairs that are present in a frame, even when their distance is nan. I have modified the Accumulator to (1) add a RAW event for every (object, hypothesis) pair with non-nan distance only, and then (2) add an extra RAW event for each object and hypothesis that was not yet added (because all their distances were nan). Roughly speaking, this reduces the number of RAW events from no*nh to no+nh. This is a big improvement:

id metrics	solver	branch	mergeOverall (sec)	wall clock time	max resident (kbytes)
no	lapsolver	develop	161.837	4:01	1405716
yes	lapsolver	develop	195.055	4:35	1765048

I think this shouldn't affect the correctness. All the tests pass (I just had to modify the tests which check the events directly).

I also modified the code that computed the IOU between all boxes to use numpy broadcasting.

This is much closer but it would be great to improve the speed even more.

[jvlmdr]

I found another big improvement. If we change the dtype of "OId" and "HId" columns from object to float, the pandas operations are much faster. However, this breaks backwards compatibility by requiring that IDs be numbers instead of strings. If we need to preserve this functionality, we could internally translate IDs to numbers. What do you think we should do @cheind ?

id metrics	solver	branch	mergeOverall (sec)	wall clock time	max resident (kbytes)
no	lapsolver	develop	9.65	1:28	1405108
yes	lapsolver	develop	19.1	1:38	1765084

[jvlmdr]

I found a few more savings. The MaskedArray data structure seems to be significantly slower. If I replace it with an explicit mask, I get another decent speed-up:

id metrics	solver	branch	mergeOverall (sec)	wall clock time	max resident (kbytes)
no	lapsolver	develop	9.492	1:01	1391496
yes	lapsolver	develop	18.954	1:11	1754388

And this might be getting into the realm of over-optimization, but if I change MOTAccumulator._events from a long list of 4-tuples to a dict with 4 long lists of scalars (and do the same for _indices), I get a small speed-up and a significant reduction in memory (since an empty tuple takes 56 bytes).

id metrics	solver	branch	mergeOverall (sec)	wall clock time	max resident (kbytes)
no	lapsolver	develop	7.353	0:58	940108
yes	lapsolver	develop	16.908	1:06	1356748

I have pushed these to my develop branch for now. Let me know what you think.

[cheind]

Jack, I don't see much of an issue with changing the ID type to float/int. No benchmark dataset that I'm aware of requires id types other than these scalars. I think if we document it properly we should be fine. Thanks Jack for your investigation!

On Fri, Jan 17, 2020 at 12:15 AM Jack Valmadre notifications@github.com wrote:

I found a few more savings. The MaskedArray data structure seems to be significantly slower. If I replace it with an explicit mask, I get another decent speed-up: id metrics solver branch mergeOverall (sec) wall clock time max resident (kbytes) no lapsolver develop 18.954 1:11 1754388

And this might be getting into the realm of over-optimization, but if I change MOTAccumulator._events from a long list of 4-tuples to a dict with 4 long lists of scalars (and do the same for _indices, I get a small speed-up and a significant reduction in memory (since an empty tuple takes 56 bytes). id metrics solver branch mergeOverall (sec) wall clock time max resident (kbytes) no lapsolver develop 16.908 1:06 1356748

I have pushed these to my develop branch for now. Let me know what you think.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cheind/py-motmetrics/issues/20?email_source=notifications&email_token=AAAJNJPZBROPLVWL2SUKEOLQ6DS7NA5CNFSM4GEE4OB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJF4OBQ#issuecomment-575391494, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJNJLPSV5L6JCODLTVJE3Q6DS7NANCNFSM4GEE4OBQ .

[jvlmdr]

Great!

[jvlmdr]

I have pushed to develop. Feel free to try again now @bbrudevold

[cheind]

Thanks Jack!

On Fri, Jan 17, 2020 at 10:36 AM Jack Valmadre notifications@github.com wrote:

I have pushed to develop. Feel free to try again now @bbrudevold https://github.com/bbrudevold

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cheind/py-motmetrics/issues/20?email_source=notifications&email_token=AAAJNJLKCLU3QHJXHV4X7WDQ6F3Y3A5CNFSM4GEE4OB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJHC6QI#issuecomment-575549249, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJNJPL75TLGLD4IBX3Z7LQ6F3Y3ANCNFSM4GEE4OBQ .

[bbrudevold]

Thanks to both of you for these improvements! I will try them out.

[cheind]

Thanks to Jack, I didn't die much :-)

Bryan B notifications@github.com schrieb am Fr., 17. Jän. 2020, 16:41:

Thanks to both of you for these improvements! I will try them out.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cheind/py-motmetrics/issues/20?email_source=notifications&email_token=AAAJNJIJUNLNUJZA4PXK4R3Q6HGULA5CNFSM4GEE4OB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJICJDA#issuecomment-575677580, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJNJLH2NQJA4FZWGAPZ7TQ6HGULANCNFSM4GEE4OBQ .

[cheind]

Do much ...

Christoph Heindl christoph.heindl@gmail.com schrieb am Fr., 17. Jän. 2020, 16:42:

Thanks to Jack, I didn't die much :-)

Bryan B notifications@github.com schrieb am Fr., 17. Jän. 2020, 16:41:

Thanks to both of you for these improvements! I will try them out.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cheind/py-motmetrics/issues/20?email_source=notifications&email_token=AAAJNJIJUNLNUJZA4PXK4R3Q6HGULA5CNFSM4GEE4OB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJICJDA#issuecomment-575677580, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJNJLH2NQJA4FZWGAPZ7TQ6HGULANCNFSM4GEE4OBQ .

[jvlmdr]

Closing now since the original issue of exaggerated memory usage is resolved.

[jvlmdr]

Hey @bbrudevold , would you mind if we added your predictions (and groundtruth) to the repository to serve as a benchmark?

[bbrudevold]

Yes that would be fine. I am glad the files have been helpful, and thanks again for the speedups!

[jvlmdr]

Sweet, thanks!