This repo contains the code of Semantics-Assisted Video Captioning Model, based on the paper "Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling Strategy". It is under review at Frontiers in Robotics and AI.
We propose three ways to improve the video captioning model. First of all, we utilize both spatial features and dynamic spatio-temporal features as inputs for semantic detection network in order to generate meaningful semantic features for videos. Then, we propose a scheduled sampling strategy which gradually transfers the training phase from a teacher guiding manner towards a more self teaching manner. At last, the ordinary logarithm probability loss function is leveraged by sentence length so that short sentence inclination is alleviated. Our model achieves state-of-the-art results on the Youtube2Text dataset and is competitive with the state-of-the-art models on the MSR-VTT dataset.
The overall structure of our model looks like this . Here is some captions generated by our model.
If you need a newer and more powerful model, please refer to Delving-Deeper-into-the-Decoder-for-Video-Captioning.
msrvttt
, msvd
, tagging
folders.cd path_to_directory_of_model; mkdir saves
run_model.sh
is used for training models and test_model.sh
is used for testing models.
Specify the GPU you want to use by modifying CUDA_VISIBLE_DEVICES
value. Specify the needed
data paths by modifying corpus
, ecores
, tag
and ref
values. The words will be sampled
by argmax strategy if argmax
is 1 and they will be sampled by multinomial strategy if argmax
is 0. name
is the name which you give to the model. test
refers to the path of the saved model
which is to be tested. Do not give a parameter to test
if you want to train a model.
bash run_model.sh
for training,
bash test_model.sh
for testing.Model | B-4 | C | M | R | Overall |
---|---|---|---|---|---|
LSTM-E | 45.3 | 31.0 | |||
h-RNN | 49.9 | 65.8 | 32.6 | ||
aLSTMs | 50.8 | 74.8 | 33.3 | ||
SCN | 51.1 | 77.7 | 33.5 | ||
MTVC | 54.5 | 92.4 | 36.0 | 72.8 | 0.9198 |
ECO | 53.5 | 85.8 | 35.0 | ||
SibNet | 54.2 | 88.2 | 34.8 | 71.7 | 0.8969 |
Our Model | 61.8 | 103.0 | 37.8 | 76.8 | 1.0000 |
Model | B-4 | C | M | R | Overall |
---|---|---|---|---|---|
v2t_navigator | 40.8 | 44.8 | 28.2 | 60.9 | 0.9325 |
Aalto | 39.8 | 45.7 | 26.9 | 59.8 | 0.9157 |
VideoLAB | 39.1 | 44.1 | 27.7 | 60.6 | 0.9140 |
MTVC | 40.8 | 47.1 | 28.8 | 60.2 | 0.9459 |
CIDEnt-RL | 40.5 | 51.7 | 28.4 | 61.4 | 0.9678 |
SibNet | 40.9 | 47.5 | 27.5 | 60.2 | 0.9374 |
HACA | 43.4 | 49.7 | 29.5 | 61.8 | 0.9856 |
TAMoE | 42.2 | 48.9 | 29.4 | 62.0 | 0.9749 |
Our Model | 43.8 | 51.4 | 28.9 | 62.4 | 0.9935 |
MSR-VTT Dataset:
Data and Models for both MSVD and MSR-VTT
MSVD Results
MSR-VTT Results
@ARTICLE{10.3389/frobt.2020.475767,
AUTHOR={Chen, Haoran and Lin, Ke and Maye, Alexander and Li, Jianmin and Hu, Xiaolin},
TITLE={A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling},
JOURNAL={Frontiers in Robotics and AI},
VOLUME={7},
PAGES={129},
YEAR={2020},
URL={https://www.frontiersin.org/article/10.3389/frobt.2020.475767},
DOI={10.3389/frobt.2020.475767},
ISSN={2296-9144},
ABSTRACT={Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.}
}