search

happyharrycn / actionformer_release

Code release for ActionFormer (ECCV 2022)
MIT License
415 stars 77 forks source link

Missing video in THUMOS14 I3D features #107

Closed Jaswar closed 1 year ago

Jaswar commented 1 year ago

Hi, thank you for this amazing contribution. I have downloaded the I3D features for the THUMOS'14 dataset using the Google Drive link. In your paper, it is mentioned there are 213 test videos and 200 validation videos. This checks out with the number of files in the i3d_features folder, as there are 413 files. However, the thumos14.json annotations file, seems to only contain 412 annotations for 212 test videos and 200 validation videos. The missing video seems to be video_test_0001292. The file for that video exists in the i3d_features folder.

Am I missing something or is the annotation for that file indeed missing? If it is missing, what was the reason for exclusion?

Thank you for your help in advance.

happyharrycn commented 1 year ago

If I recall correctly, that missing video does not contain any annotated events, and is thus excluded. You can double check this using the official annotations of THUMOS'14 dataset.

Jaswar commented 1 year ago

Indeed, having checked the annotations from the original file, this video does not appear in any action class. Thank you for your help.

I have however, a follow up question. If I'm understanding the codebase correctly, wouldn't excluding this video from the annotations file also exclude it from testing (as I believe they are similarly excluded in ActivityNet #57)? If so, why exactly is that done? Wouldn't it also make sense to test on videos that contain no action? On the other hand, I do realize the difference would be negligible even if those are included.

happyharrycn commented 1 year ago

Ideally, a model should be tested on all videos, including those without actions. Testing on "empty" videos provides a way to evaluate false positive rates. Digging into our notes, that video in THUMOS'14 (video_test_0001292) appears to be a video that does contain target action categories (golf swing), yet was not annotated. We have thus decided to exclude this video from evaluation, following the standard practice in the community.

Like you mentioned, the difference would be negligible even if this video is included.