It is a re-implementation code for the following paper:
Kao Zhang, Zhenzhong Chen. Video Saliency Prediction Based on Spatial-Temporal Two-Stream Network. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), vol. 29, no. 12, pp. 3544-3557, 2019.
[Online] Avaliable: https://ieeexplore.ieee.org/document/8543830
The code was developed using Python 3.6 & Keras 2.2.4 & CUDA 9.0. There may be a problem related to software versions.To fix the problem, you may look at the implementation in "zk_models.py" file and replace the syntax to match the new keras environment.
Download the pre-trained models and put the pre-trained model into the "Models" file.
TwoS-model Baidu Drive; Google Drive; OneDrive (210M)
SF-Net-model Baidu Drive; Google Drive; OneDrive (58M)
Currently, the code supports python 3.6
please change the working directory: "wkdir" to your path in the "zk_config.py" file, like
dataDir = 'E:/Code/IIP_TwoS_Saliency/DataSet'
More parameters are in the "zk_config.py" file.
Run the demo "Test_TwoS_Net.py" and "Train_TwoS_Net.py" to test or train the model.
The full training process:
Our model is trained on SALICON and part of the DIEM dataset. We train the SF-Net in spatial stream based on the pre-trained VGG-16 model and the training set of SALICON dataset. Then, we train the whole network on the training set of DIEM dataset, and fix the parameters of the trained SF-Net.
And it is easy to change the output format in our code.
If you use the TwoS video saliency model, please cite the following paper:
@article{Zhang2018Video,
author = {Kao Zhang and Zhenzhong Chen},
title = {Video Saliency Prediction Based on Spatial-Temporal Two-Stream Network},
journal = {IEEE Transactions on Circuits and Systems for Video Technology },
year = {2018}
}
Kao ZHANG
Laboratory of Intelligent Information Processing (LabIIP)
Wuhan University, Wuhan, China.
Email: zhangkao@whu.edu.cn
Zhenzhong CHEN (Professor and Director)
Laboratory of Intelligent Information Processing (LabIIP)
Wuhan University, Wuhan, China.
Email: zzchen@whu.edu.cn
Web: http://iip.whu.edu.cn/~zzchen/