Does it only support the gt_segment that just has one action for one moment?

[ttgeng233]

Thank you for your good project! in lines 510-515 at ./libs/modeling/meta_archs.py, I can't understand the code but it seems to keep only one action for one moment in gt_reg, but in experiments, I found that EPIC-kitchens-100 dataset has many videos that have more than one action at the same time, as shown in its original paper: So, if there are some errors in the code, how to modify it to allow multiple actions per timestep?

[happyharrycn]

Our code allows a single moment to be matched to multiple overlapping GT segments. L517-526 in the same file handles this.

[tzzcl]

Hi, I saw your issues #43 before. My answer currently remains the same. The code you mentioned is that each feature point will only have one regression target. And ActionFormer will perform class-agnostic boundary regression.

For EPIC-Kitchens 100, the action is composed by a verb plus a noun. Though it will have some overlap between different actions, since our center sampling strategy, we can still achieve good performance on EPIC-Kitchens dataset since most feature point (which represents the center area of an action) will only have one class.

If you want to tweak ActionFormer for multiclass datasets, you may need to perform class-aware regression, i.e., in the code you mentioned, you keep the regression length for multiple length, and perform per-class boundary regression.

[happyharrycn]

OK. I see the confusion here. In short, I don't think there is a need for class-aware regression. An important detail is that when decoding the action instances, our code considers every category in each feature slot as a candidate.

To further clarify, let us consider the following cases.

1. There are multiple GT segments that are not overlapping. This will not bring any issue.
2. There are multiple GT segments with minor overlap. Each segment will be assigned to a different feature slot, as only those slots around the center of a segment are considered as positive. Again, there is no issue here.
3. There are multiple GT segments with major overlap, where multiple segments might be assigned to the same set of feature slots. Our current implementation will consider a multi-class classification for those feature slots, yet will only regress the closest temporal boundary for each slot. At inference time, if some segments have major overlap, our model will predict one of the boundaries yet multiple categories at the same slot, leading to multiple detected action instances with the same temporal span. This should be totally fine in practice, as again, the boundaries of those segments are similar anyway.
4. There are multiple GT segments with the same span. This is the case on THUMOS'14 and is handled similar to 3.

[tzzcl]

For case 3, a major failure mode will lies in the sub-action action pattern, i.e., one sub-action is localized in the center of another action.

For example, one action is composed by three subactions, and the second subaction is localized at the center of the action. In that case, ActionFormer will fail. These cases can be found in complex datasets like FineGym.

[happyharrycn]

In the example you mentioned, the second subaction, co-centered with the action, is likely distributed to a different level on FPN. A corner case will be that two GT segments are co-centered on the same FPN level. But again, predicting any of the two boundaries will lead to a pretty high tIoU (>0.5) for the other, due to the design of the FPN.

[tzzcl]

closed due to inactivity.