two-stream-fusion-for-action-recognition-in-videos

We have implemented convolutional two stream network for action recognition for two cases -

• Two stream average fusion at softmax layer.
• Two stream fusion at convolutional layer.

  1. Data

  We have used UCF101 dataset for this project. For utilization of temporal information, we have used optical flow images and RGB frames for utilizing spatial information. The pre-processed RGB frames and flow images can be downloaded from feichtenhofer/twostreamfusion)

• RGB images

  wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.001
  wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.002
  wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.003
  
  cat ucf101_jpegs_256.zip* > ucf101_jpegs_256.zip
  unzip ucf101_jpegs_256.zip

• Optical Flow

  wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.001
  wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.002
  wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.003
  
  cat ucf101_tvl1_flow.zip* > ucf101_tvl1_flow.zip
  unzip ucf101_tvl1_flow.zip

  For both the cases we have used a stack of 10 RGB frames as input for spatial stream and a stack of 50 optical flow frames as input for temporal stream. So, for a batch size = 4, typical spatial loader will look like the image below -

2. Models

We have used vgg-19 model pre-trained on ImageNet for both the streams.

3. Implementation details for both cases

• Note :- To do weight transformation for first layers of ConvNets, we first average the weight value across the RGB channels and replicate this average value by the channel number in that ConvNet.

  3.1 Two stream average fusion at softmax layer

  The architecture for this case has been shown in the figure below -

  3.2 Two stream fusion at convolution layer

  The Architecture for this case is shown in the Figure below - The ConvNets are being replaced be vgg model, trained on ImageNet.

  4. Training Models

• Please modify this path and this path to fit the UCF101 dataset on your device.
• If you want to change the number of frames in RGB stack, then modify here. Select the frames you want to have in the stack. If you want, you can also introduce randomness in choosing the frames for stacking.

5. Performance

5.1 Performance for two stream average fusion

• For first 20 classes of UCF101 dataset

  Network	Acc.
  Spatial cnn	91.96%
  Motion cnn	97.30%
  Average fusion	99.01%

• For all 101 classes of UCF101 dataset

  Network	Acc.
  Spatial cnn	48.64%
  Motion cnn	51.17%
  Average fusion	62.13%

  5.2 Performance for two stream fusion at convolution layer

• For first 20 classes of UCF101 dataset, we get an accuracy of 96.01 %

• For all 101 classes of UCF 101 dataset, we get an accuracy of 68.23 %

6. Reference Paper

• [1] Two-stream convolutional networks for action recognition in videos
• [2] Convolutional Two-Stream Network Fusion for Video Action Recognition