to train optical flow branch

[lhoangan]

I cannot find a script to train the optical flow branch. I am wondering if you could release the data layer and train.prototxt for that?

[JingchunCheng]

You can train flow branch with your own flow data layer according to the optical flow dataset. For example, on SINTEL dataset, my data layer for optical flow is very similar to that of DAVIS segmentation, except that I changed the format of SINTEL's flow annotation from '.flo' to '.mat' and load it by scipy.io.loadmat.

[lhoangan]

So the data layer (and augmentation) would stay the same and only has the input statements changed to read flow format? About the train prototxt file, what should I change? From the paper, it says to freeze the segmentation branch and train the flow branch, so I've put

params {
    lr_mult: 0
    decay_mult: 0}

on all the res layers, while setting lr_mult and decay_mult on flowS layers to one in FlowNetS paper. Is that the correct setting?

I found some layers in score_segmentation and flow_segmentation that also have lr_mult set to 0. Should I do anything with them?

Besides, what are the learning rates for flow training? The paper said 10e-8 for segmentation and 10e-9 for flow, but in solver.prototxt it says 1e-9 for segmentation already.

[JingchunCheng]

I only used the "flip" augmentation in optical flow branch.

For training of optical flow branch, you need to remove the "lr_mult: 0" in FlowNetS layers and add "propagation_down: false" in ResNet conv3,4,5 to freeze parameters in segmentation branch.

The learning rate varies from 1e-6 to 1e-10 for different datasets and settings, you can test and choose a proper one according to training loss.

[lhoangan]

Thanks for your information. I'll try it and will get back when it starts working. Could you also tell how much should we expect the flow loss? I've tried before with my setting and it's fluctuated around 5e6 to 5e7. Isn't it a bit too big for a loss?

[JingchunCheng]

The training loss for segmentation is very large for that we did not do normalization in the prediction layer. The learning rate is set to be 1e-8~1e-10 because of this large loss. When training the segmentation branch, it's fine as long as the accuracy curve goes up.

For optical flow, our training loss on SINTEL drops from about 60 to around 0.8. You can start with learning rate 1e-5 or 1e-6 and see whether the loss curve is going down.

[lhoangan]

That's flow loss of 60 is astonishing to me. I am using the loss layer from FlowNetS architecture:

layer {                                                                         
   name: "flow_loss2"                                                            
   type: "L1Loss"                                                                
   bottom: "score_flow"                                                          
   bottom: "label"                                                               
   top: "flow_loss"                                                              
   l1_loss_param {                                                               
      l2_per_location: true                                                       
   }
}   

and it gives me the loss for flow branch around 3e+06 and keeps fluctuating around that (not decreasing, but sometimes going up to 5e+07) . My learning rate is around 1e-11 (the network overshoots and gives nan if base_lr is higher than 1e-10)

One more point: FlowNetS does use multiple losses at different scales in the decoder; is it the case of SegFlow?

[JingchunCheng]

I'm not sure what caused your flow loss to be that large. Maybe you can refer to our training prototxt at the following link.

Link: https://pan.baidu.com/s/1SO_rPCUeG-mjQ_wsKH4NOQ Password: p75d