SlowFast feature extraction

[Guilleuz]

Hi!, thanks for the great work.

I was wondering if you could help me out with the feature extraction from the SlowFast model, as I have a dataset very similar to Epic Kitchens and I was trying to train and evaluate ActionFormer with it. I have tried using some tools (Gluoncv and SlowFast Feature Extractor) but when training using the features obtained the scores where really low, not much higher than 0% for both nouns and verbs.

Thank you in advance.

[tzzcl]

For your question, I think you may check the fps and feat_stride config in the config file according to your feature extraction settings.

[Guilleuz]

Hello, the features where extracted from 32-frame clips at 25 fps, using an 8-frame temporal stride. I tried the model using those same values for num_frames, default_fps and feat_stride but the scores were just as low as before.

[tzzcl]

For your questions, it may come from multiple sides:

1. The feature extraction part, you may use a classification task to verify your pre-trained models are correctly loaded.
2. I think you may check your training log to find if the model does not converge or have other issues.

[happyharrycn]

Extracting video features could be a bit tricky. I will list a few potential pitfalls for your reference. If further help is needed, feel free to send us an email.

• Input format: Are video frames properly decoded and converted for model input? Common issues include (a) frames are decoded with less visual fidelity (say lower resolution than the model input, or a low bitrate), (b) frame rate does not quite match the model input (e.g, the vanilla SlowFast expects 30 FPS and samples 64 frames from every other frame. Our default parameters were based on a fine-tuned version of SlowFast (and no the vanilla one), and (c) input normalization was not handled properly (e.g., order of color channels).
• Input resolution: How are the decoded frames feed into the model? SlowFast is fully convolutional, allowing one to feed inputs with slightly different resolutions (in comparison to training). If you are dealing with 16:9 videos (as for EPIC-Kitchens), the standard center crop will not work very well. We instead pass all pixels into the model.
• Pre-training: If your dataset is quite different from the pre-training data (Kinetics), the video backbone is less likely to work. Again, we considered a SlowFast model that is further fine-tuned in EPIC-Kitchens.

[Guilleuz]

Thank you both! I will look into it and see if I can get better scores.

[happyharrycn]

I will mark this as closed. Feel free to re-open this issue or send us an email if this is not resolved.

[emptybird]

Hello, the features where extracted from 32-frame clips at 25 fps, using an 8-frame temporal stride. I tried the model using those same values for num_frames, default_fps and feat_stride but the scores were just as low as before.

May I inquire how you utilized slowfast_feature_extractor to implement custom temporal stride?