Evaluation Code for CroHD

[HarryHsing]

Hi, Dr. Harley

I re-test the new reference model (updated on 12/15/22) on CroHD, and here is the result I got: ate = 7.71; ate_vis = 5.79; ate_occ = 11.12 The result is different from the one on the github homepage: vis: 4.57, occ: 7.71 It seems that 7.71 is the result averaged over vis. and occ.

This is the original line of code, I modified it a little bit to print vis. and occ. results: https://github.com/aharley/pips/blob/d8d77e2c675388e71c0729ba928f564de31a4c23/test_on_crohd.py#L346-L348

[aharley]

Yes, that's not an error I think, but definitely could be written better. The "real" vis/occ switch is in req_occlusion=True: https://github.com/aharley/pips/blob/d8d77e2c675388e71c0729ba928f564de31a4c23/test_on_crohd.py#L233 The vis/occ separation you are referencing has to do with the per-timestep occupancy, whereas this binary variable selects data where occlusions happen or not.

Using average ate is indeed intentional, since it gives us the accuracy on tracks that undergo occlusion. So, to reproduce the repo's vis/occ results (vis: 4.57, occ: 7.71), you need to run the model twice. (I should probably write that part of the code better...)

[HarryHsing]

Yes, that's not an error I think, but definitely could be written better. The "real" vis/occ switch is in req_occlusion=True:

https://github.com/aharley/pips/blob/d8d77e2c675388e71c0729ba928f564de31a4c23/test_on_crohd.py#L233

The vis/occ separation you are referencing has to do with the per-timestep occupancy, whereas this binary variable selects data where occlusions happen or not. Using average ate is indeed intentional, since it gives us the accuracy on tracks that undergo occlusion. So, to reproduce the repo's vis/occ results (vis: 4.57, occ: 7.71), you need to run the model twice. (I should probably write that part of the code better...)

Got it, Dr. Harley. I re-run the code, and now the result can be matched. Thanks again for your support!