search

happyharrycn / actionformer_release

Code release for ActionFormer (ECCV 2022)
MIT License
434 stars 77 forks source link

The meaning of label '4' #10

Closed Z2HENG closed 2 years ago

Z2HENG commented 2 years ago

Thanks for your nice work! In the released code I notice that one action label '4' is masked in the Thumos14 dataset. I wonder why the action category should be masked. What's the purpose of this step ?

happyharrycn commented 2 years ago

Good question! In THUMOS14 dataset, the action category "Cliff Diving" (ID: 4) is a subset of the category "Diving" (ID: 7). Namely, all "Cliff Diving" instances are also annotated as "Diving", yet not all "Diving" instances are marked as "Cliff Diving". This brings an issue for the implementation of label assignment during training.

Specifically, our current implementation assumes that a moment can be matched to at most one ground truth (GT) action instance, and thus could not handle the case where there are two GT action instances with exactly the same temporal boundaries (yet different labels).

Our solution, somewhat hacky, is to remove all instances from the action category "Cliff Diving" during training by masking out those instances. Note that (1) all instances from "Diving" still remain; and (2) this is only done during training. At test time, our model is still evaluated on all categories. The downside of this solution is that our model will not be able to recognize "Cliff Diving" on the test set, as it never saw an training sample from this category. Part of this discrepancy is resolved by fusing external classification scores. To verify this, you can disable score fusion and print the mAP scores per category, and the mAP for "Cliff Diving" (ID: 4) should be close to 0.

happyharrycn commented 2 years ago

Let me know if further clarification is needed. Otherwise, I will mark this issue as resolved.

Z2HENG commented 2 years ago

Let me know if further clarification is needed. Otherwise, I will mark this issue as resolved.

Good question! In THUMOS14 dataset, the action category "Cliff Diving" (ID: 4) is a subset of the category "Diving" (ID: 7). Namely, all "Cliff Diving" instances are also annotated as "Diving", yet not all "Diving" instances are marked as "Cliff Diving". This brings an issue for the implementation of label assignment during training.

Specifically, our current implementation assumes that a moment can be matched to at most one ground truth (GT) action instance, and thus could not handle the case where there are two GT action instances with exactly the same temporal boundaries (yet different labels).

Our solution, somewhat hacky, is to remove all instances from the action category "Cliff Diving" during training by masking out those instances. Note that (1) all instances from "Diving" still remain; and (2) this is only done during training. At test time, our model is still evaluated on all categories. The downside of this solution is that our model will not be able to recognize "Cliff Diving" on the test set, as it never saw an training sample from this category. Part of this discrepancy is resolved by fusing external classification scores. To verify this, you can disable score fusion and print the mAP scores per category, and the mAP for "Cliff Diving" (ID: 4) should be close to 0.

Thanks for your detailed explanation! This solution is reasonable and effective. Nice work!