question of FlownetS

[zmlshiwo]

Hello, I am a bit puzzled by the steps to predict the flow after the image is extracted

layer {
  name: "Convolution1"
  type: "Convolution"
  bottom: "conv6_1"
  top: "predict_flow6"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 1
    decay_mult: 0
  }
  convolution_param {
    num_output: 2
    pad: 1
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
    engine: CUDNN
  }
}

The input is feature map of two input image, and then convolution....the output is predict_flow. How this layer is converted from the feature map to the optical flow？ And the num_out = 2 is the U and V of predict_flow?

[nikolausmayer]

Hi,

you are assuming a difference where none exists. The network is trained to produce optical flow in this layer, so while the output is technically a "feature map", those features are not abstract or "deep" — the features are the optical flow.

(Note that this only applies early in the training; later on, the Convolution1 layer is no longer trained to produce flow and will do something else, but the argument is valid for all layers which produce predict_flow* outputs)

num_output: 2 refers to the two flow channels, yes.

[zmlshiwo]

OK, i understand. And if i want to train flownet2, How much memory of GPU do I need at least? I have a 4GB GPU. And i have not found the flownet+v model mentioned in flownet1.0, the v is “variation”. Thank you! @nikolausmayer

[eddy-ilg]

We trained FlowNet2 on TITANX cards (12GB). Altough I am not sure if it might also work with 8GB. flownet+v is with variational refinement, but this refinement is not published.