Closed amanchadha closed 2 years ago
Hi 👋 ! Indeed!
I am afraid, I don't have the exact snippet which does .mp4
-> .npy
-> .hdf5
. However, the procedure is quite straightforward. Someone asked it before and I wrote one from my memory: https://github.com/v-iashin/MDVC/issues/11#issuecomment-645371791. What you need there is the answer to the first question. I didn't run it back then. I just wrote it down in my comment. So, please ensure it won't fail with some errors (check out the following comments on possible bugs). Overall, just extract features with video_features
(repo) code and run that script on top of it.
Are the audio and video features aligned by time in the hdf5 file? Is that what T_audio/T_video stands for?
I am not sure what you mean here. T_audio
and T_video
are temporal dims of features. The I3D
features are extracted from 24 frames from a video sampled at 25 fps and temporarily span 0.96 (24/25) seconds with no overlap. At the same time, VGGish
features are extracted from 0.96 s audio segments. As you see, both sequences should have the same temporal length if they are extracted from the same video. Therefore, they are aligned. However, sometimes (I don't remember how rarely), they are not equal. In this case, we just trim the longest one to match the shortest one (see the i3d
part in the snippet you provided).
Is the D_audio/D_video simply the feature dimension?
Yes, it is.
Hi Vladimir,
Do the audio features for each video cover the entire video? Did you filter out the audio segments that are not inside event proposals?
Thank you!
Yes, similar to visual features and speech, the audio is available for the entire video. And yes we trim the modalities to be in a segment as shown here
Hi Vladimir,
Long time no talk :) I was wondering if you can share the code that converted the .npy features (from your VGGish and I3D feature extractor) that you made available to me mid last year, to .hdf5 in MDVC Readme: Usage. In particular, I am interested in understanding how you "align" the audio and video features (based on the code below).
Questions:
Thanks again for your help!