search

sampepose / flownet2-tf

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
MIT License
404 stars 195 forks source link

Problem when using flow_warp for backward pass #4

Closed jgj1986 closed 7 years ago

jgj1986 commented 7 years ago

When reading the code about FlowNet_CS, I found that the train() in net.py call loss() function, which only call self.net_s.loss, without self.net_c.loss(). So the self.net_c CANNOT be trained when training the FlowNet_CS, the code use trained FlowNet_C model as self.net_c for FlowNet_CS. Right?

You kown that the number of dataset with flow is very small, so i want the train the net with warped the images, such as:

    #... code about the flownet_c net
    """ END: Refinement Network """
    flow = predict_flow2 * 20.0
    flow = tf.image.resize_bilinear(flow, tf.stack([height, width]), align_corners=True)                                                   
    warped = flow_warp(inputs['input_b'], flow)

    return {'input_b': warped, 'input_a': inputs['input_a']}

def loss(self, flow, predictions):
    tf.losses.add_loss(average_endpoint_error(predictions['input_a'], predictions['input_b']))
    return tf.losses.get_total_loss()

This code need to use _flow_warp_grad(), but I got errors when running it:

    ...
    tensorflow.python.framework.errors_impl.InternalError: Missing 1-th output from gradients/FlowNetC/FlowWarp_grad/FlowWarpGrad = FlowWarpGrad[_device="/job:localhost/replica:0/task:0/gpu:0"](ResizeBicubic_1/_523, FlowNetC/ResizeBilinear, gradients/average_endpoint_error/Sub_grad/tuple/control_dependency)
    [[Node: train_op/control_dependency/_579 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_4895_train_op/control_dependency", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

So, does flow_warp allow a backward pass while training? or any problem in my code ? Thanks!

sampepose commented 7 years ago

The reasoning for calling only self.net_s.loss is because by definition, FlowNet-CS is created by: 1) training FlowNet-C, 2) fixing the FlowNet-C weights and appending a randomly initialized FlowNet-S, 3) training the FlowNet-S part of this newly created FlowNet-CS

It will not necessarily produce the same results to train CS in one go. If you want to do this, you should create a custom CS class based off of FlowNetCS with the correct loss.

The flow warp operation does have a defined gradient, so the backwards pass should work. Make sure you're on the latest version of the code since I fixed this recently. Your code looks correct.

happyfootgogo commented 7 years ago

I have the same problem. Is there any possible solution for this error?

sampepose commented 7 years ago

@happyfootgogo, please make sure you have the latest version of the code. If you still have this issue, please make a separate issue with all of the details.