The TGIF-QA dataset contains 165K QA pairs for the animated GIFs from the TGIF dataset [Li et al. CVPR 2016]. The question & answer pairs are collected via crowdsourcing with a carefully designed user interface to ensure quality. The dataset can be used to evaluate video-based Visual Question Answering techniques.
In this page, you can find the code and the dataset for our IJCV journal article.
- Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim and Gunhee Kim. Video Question Answering with Spatio-Temporal Reasoning. IJCV, 2019. [Journal Link]
Please check this tag if you are interested in our CVPR 2017 setting.
The code and the dataset are free to use for academic purposes only. If you use any of the material in this repository as part of your work, we ask you to cite:
@article{jang-IJCV-2019,
author = {Yunseok Jang and Yale Song and Chris Dongjoo Kim and Youngjae Yu and Youngjin Kim and Gunhee Kim},
title = {{Video Question Answering with Spatio-Temporal Reasoning}}
journal = {IJCV},
year = {2019}
}Note: Since our CVPR 2017 paper, we extended our dataset by collecting more question and answer pairs (the total count has increased from 104K to 165K) and re-ran experiments with the new dataset. The journal article and the arXiv paper is the most update one.
Have any question? Please contact:
Yunseok Jang (yunseok.jang@snu.ac.kr), Chris Dongjoo Kim (cdjkim@vision.snu.ac.kr), and Yale Song (yalesong@microsoft.com)
Q&A Types and Examples
| Q&A Type | Repetition Count | Repeating Action | State Transition | Frame QA |
|---|---|---|---|---|
| Visual Input (GIF) |  |
 |
 |
 |
| Question | How many times does the cat lick? | What does the cat do 3 times? | What does the model do after lower coat? | What is the color of the bulldog? |
| Answer | 7 times | Put head down | Pivot around | Brown |
# Q&A Pairs
| Task | Train | Test | Total |
|---|---|---|---|
| Repetition Count | 26,843 | 3,554 | 30,397 |
| Repeating Action | 20,475 | 2,274 | 22,749 |
| State Transition | 52,704 | 6,232 | 58,936 |
| Frame QA | 39,392 | 13,691 | 53,083 |
| Total | 139,414 | 25,751 | 165,165 |
Quantitative Results
| Model | Repetition Count (L2 loss) | Repeating Action (Accuracy) | State Transition (Accuracy) | Frame QA (Accuracy) |
|---|---|---|---|---|
| Random Chance | 19.62 | 20.00 | 20.00 | 0.06 |
| Most Frequent words | 7.78 | 31.40 | 30.05 | 17.49 |
| VIS+LSTM (aggr) [NIPS 2015] | 5.09 | 46.84 | 56.85 | 34.59 |
| VIS+LSTM (avg) [NIPS 2015] | 4.81 | 48.77 | 34.82 | 34.97 |
| VQA-MCB (aggr) [EMNLP 2016] | 5.17 | 58.85 | 24.27 | 25.70 |
| VQA-MCB (avg) [EMNLP 2016] | 5.54 | 29.13 | 32.96 | 15.49 |
| CT-SAN [CVPR 2017] | 5.14 | 56.14 | 63.95 | 39.64 |
| Co-Memory [CVPR 2018] | 4.10 | 68.20 | 74.30 | 51.50 |
| ST-VQA (Ours) | 4.22 | 73.48 | 79.72 | 51.96 |
Qualitative Results
Spatial Attention
Temporal Attention
The red dotted boxes over heatmaps indicate segments in a video that include the ground-truth answers.
Attentions Visualized in Time
The yellow bar indicates the strength of temporal attention at the visualized time.
| Q&A Type | Repetition Count | Repeating Action | State Transition | Frame QA |
|---|---|---|---|---|
| Lively Visual (GIF) |  |
 |
 |
 |
| Question | How many times does the man shave chest ? | What does the boy do 3 times ? | What does the man do before kiss toy ? | What are the group of boys singing , dancing , and playing ? |
| Answer | 2 times | Wave hands | Pet toy | Instruments |
| Lively Visual (GIF) |  |
 |
 |
 |
| Question | How many times does the man flip circle ? | What does the behind do 3 times ? | What does the woman do after raise leg ? | What is the color of the shirt ? |
| Answer | 2 times | Shake butt | Kick a mug | White |
Notes
Last Edit: May 22, 2020