search

microsoft / SwinBERT

Research code for CVPR 2022 paper "SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning"
https://arxiv.org/abs/2111.13196
MIT License
237 stars 34 forks source link

Trimming the YouCook2 videos + "./datasets/YouCook2/yc2_subtitles.jsonl"? #15

Open movinghoon opened 2 years ago

movinghoon commented 2 years ago

Thanks for your awesome work. I have two questions about running the code on the YouCook2 dataset.

(1) It seems that to run on the YouCook2 dataset, we need to trim the downloaded YC2 videos with segments in "youcookii_annotations_trainval.json", right? (i.e., GLd3aX16zBg -> GLd3aX16zBg_0, GLd3aX16zBg_1, GLd3aX16zBg_2, GLd3aX16zBg_3, GLd3aX16zBg_4) Can you share the method of how you did it? I have tried "ffmpeg -i input.mp4 -ss $START_TIME -to $END_TIME -c copy trim.mp4", but the command converts some frames into black images.

(2) Where can we get "./datasets/YouCook2/yc2_subtitles.jsonl" in "prepro/tsv_preproc_youcook2.py"? I couldn't find it on the YC2 website.

Yuhan-Shen commented 2 years ago

For the YouCook2 subtitles, I got it from VALUE leaderboard via the link https://datarelease.blob.core.windows.net/value-leaderboard/yc2r/yc2_subtitles.jsonl Not sure if the authors used the same file though.

JoseponLee commented 2 years ago

Can you please share the trimmed videos of youcook2? It will be very helpful. Thank you very much. My email is lijiapeng@stu.xjtu.edu.cn

SuleBai commented 1 year ago

Thanks for your awesome work. I have two questions about running the code on the YouCook2 dataset.

(1) It seems that to run on the YouCook2 dataset, we need to trim the downloaded YC2 videos with segments in "youcookii_annotations_trainval.json", right? (i.e., GLd3aX16zBg -> GLd3aX16zBg_0, GLd3aX16zBg_1, GLd3aX16zBg_2, GLd3aX16zBg_3, GLd3aX16zBg_4) Can you share the method of how you did it? I have tried "ffmpeg -i input.mp4 -ss $START_TIME -to $END_TIME -c copy trim.mp4", but the command converts some frames into black images.

(2) Where can we get "./datasets/YouCook2/yc2_subtitles.jsonl" in "prepro/tsv_preproc_youcook2.py"? I couldn't find it on the YC2 website.

@movinghoon Hi, did you figure out how to trim the downloaded YC2 videos with segments in "youcookii_annotations_trainval.json"? Thanks a lot.

AdrienneDeganutti commented 11 months ago

Hi @movinghoon, @Mythszj,

I wasn't able to find how the authors generated the trimmed YC2 videos for the training and validation splits, but I worked around this by using the youcookii_annotations_trainval.json file and splitting each video into their respective clips by referring to the "segment": [start, end]" values.

To do this I converted the segment values into timecode format by calculating the relation between the raw video's duration, and the video duration given by youcookii_annotations_trainval.json.

I'm not sure if this is how the authors processed the clips, but so far it has worked for me!