search

linjieli222 / HERO_Video_Feature_Extractor

Video Feature Extraction Code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training"
https://arxiv.org/abs/2005.00200
MIT License
97 stars 13 forks source link

Extract features from frames #1

Closed shinying closed 3 years ago

shinying commented 3 years ago

Hi @linjieli222,

Thank you for releasing this feature extractor. Though it is well designed for processing videos, I only have frames as inputs, similar to TVQA (3 fps). I'm wondering if you have any suggestion on setting parameters and modifying video_loader.

I look forward to your reply.

Shinying

linjieli222 commented 3 years ago

Hi Shinying,

Thanks for your interests in this project. Our video_loader is designed to take in original video files. Moreover, the original slowfast codebase requires continuous video frames with high framerate. It would be the best if you can obtain the original raw video files. If not, a simple way that I can think of is to repeat the frames to fake the high framerate, but one can imagine that the information carried in the repeated frames is duplicated. For loading the input video frames, you probably don't need ffprobe anymore.

Thanks, Linjie

shinying commented 3 years ago

Hi @linjieli222

Thanks for your reply and help. I'll try some different techniques.

Shinying