Flownet2CSS weight mismatch issue

[theRain0450]

Hello, first of all, thank you very much for your excellent work content. However, I am having some weight mismatch issues when trying to generate optical flow files using the Flownet2CSS pre-trained weights you provided, the error is reported as follows:

Traceback (most recent call last):
  File "main.py", line 214, in <module>
    model_and_loss.module.model.load_state_dict(checkpoint['state_dict'])
  File "/home/dyq/miniconda3/envs/flownet2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for FlowNet2CSS:
        Unexpected key(s) in state_dict: "flownets_d.conv0.0.weight", "flownets_d.conv0.0.bias", "flownets_d.conv1.0.weight", "flownets_d.conv1.0.bias", "flownets_d.conv1_1.0.weight", "flownets_d.conv1_1.0.bias", "flownets_d.conv2.0.weight", "flownets_d.conv2.0.bias", "flownets_d.conv2_1.0.weight", "flownets_d.conv2_1.0.bias", "flownets_d.conv3.0.weight", "flownets_d.conv3.0.bias", "flownets_d.conv3_1.0.weight", "flownets_d.conv3_1.0.bias", "flownets_d.conv4.0.weight", "flownets_d.conv4.0.bias", "flownets_d.conv4_1.0.weight", "flownets_d.conv4_1.0.bias", "flownets_d.conv5.0.weight", "flownets_d.conv5.0.bias", "flownets_d.conv5_1.0.weight", "flownets_d.conv5_1.0.bias", "flownets_d.conv6.0.weight", "flownets_d.conv6.0.bias", "flownets_d.conv6_1.0.weight", "flownets_d.conv6_1.0.bias", "flownets_d.deconv5.0.weight", "flownets_d.deconv5.0.bias", "flownets_d.deconv4.0.weight", "flownets_d.deconv4.0.bias", "flownets_d.deconv3.0.weight", "flownets_d.deconv3.0.bias", "flownets_d.deconv2.0.weight", "flownets_d.deconv2.0.bias", "flownets_d.inter_conv5.0.weight", "flownets_d.inter_conv5.0.bias", "flownets_d.inter_conv4.0.weight", "flownets_d.inter_conv4.0.bias", "flownets_d.inter_conv3.0.weight", "flownets_d.inter_conv3.0.bias", "flownets_d.inter_conv2.0.weight", "flownets_d.inter_conv2.0.bias", "flownets_d.predict_flow6.weight", "flownets_d.predict_flow6.bias", "flownets_d.predict_flow5.weight", "flownets_d.predict_flow5.bias", "flownets_d.predict_flow4.weight", "flownets_d.predict_flow4.bias", "flownets_d.predict_flow3.weight", "flownets_d.predict_flow3.bias", "flownets_d.predict_flow2.weight", "flownets_d.predict_flow2.bias", "flownets_d.upsampled_flow6_to_5.weight", "flownets_d.upsampled_flow6_to_5.bias", "flownets_d.upsampled_flow5_to_4.weight", "flownets_d.upsampled_flow5_to_4.bias", "flownets_d.upsampled_flow4_to_3.weight", "flownets_d.upsampled_flow4_to_3.bias", "flownets_d.upsampled_flow3_to_2.weight", "flownets_d.upsampled_flow3_to_2.bias", "flownetfusion.conv0.0.weight", "flownetfusion.conv0.0.bias", "flownetfusion.conv1.0.weight", "flownetfusion.conv1.0.bias", "flownetfusion.conv1_1.0.weight", "flownetfusion.conv1_1.0.bias", "flownetfusion.conv2.0.weight", "flownetfusion.conv2.0.bias", "flownetfusion.conv2_1.0.weight", "flownetfusion.conv2_1.0.bias", "flownetfusion.deconv1.0.weight", "flownetfusion.deconv1.0.bias", "flownetfusion.deconv0.0.weight", "flownetfusion.deconv0.0.bias", "flownetfusion.inter_conv1.0.weight", "flownetfusion.inter_conv1.0.bias", "flownetfusion.inter_conv0.0.weight", "flownetfusion.inter_conv0.0.bias", "flownetfusion.predict_flow2.weight", "flownetfusion.predict_flow2.bias", "flownetfusion.predict_flow1.weight", "flownetfusion.predict_flow1.bias", "flownetfusion.predict_flow0.weight", "flownetfusion.predict_flow0.bias", "flownetfusion.upsampled_flow2_to_1.weight", "flownetfusion.upsampled_flow2_to_1.bias", "flownetfusion.upsampled_flow1_to_0.weight", "flownetfusion.upsampled_flow1_to_0.bias". 

Personally, I think this is because the weights are not the model weights of Flownet2CSS, because after I changed the model to Flownet2, the program ran smoothly to generate the optical flow file. Also in the weights you provided, the name of the weight is written as FlowNet2 instead of FlowNet2CSS I am confused about this, are you providing the wrong weights? Or I didn't find the right way to use it?

Lastly, I wish you the best of luck with your work and look forward to hearing from you!

[theRain0450]

@hzxie

[hzxie]

I don't think I included the code and pretrained model for FlowNet2 in this repository. You should submit an issue in the FlowNet2 repository.

[theRain0450]

I don't think I included the code and pretrained model for FlowNet2 in this repository. You should submit an issue in the FlowNet2 repository.

No, I think you may have misunderstood me. I mean, I have tried to download the pre-training weights you provided, but this weight doesn't work on the Flownet2CSS mod, he can only run it on Flownet2, and I tried to run

python main.py --inference --model FlowNet2 \
--resume /home/dyq/PythonProject/flownet2-pytorch-master/saves/FlowNet2_checkpoint.pth.tar \

That's correct, but this --model FlowNet2 doesn't look like it uses Flownet2CSS , and when I switch to the --model FlowNet2CSS program it reports an error that the weights don't match. I think maybe the weights you provided are for the FlowNet2 model and not the FlowNet2CSS model?

[hzxie]

There is no script named main.py. The pretrained weights is not provided by me but NVIDIA.

[theRain0450]

In this script. There is no script named main.py. The pretrained weights is not provided by me but NVIDIA.

Thank you very much for your answer! This main.py script comes with Flownet2 and I want to run this script to generate flo optical flow files. Now my aim is to generate DAVIS photostream files to be stored in the OpticalFlows folder. If the weights are not provided by you, can I do it this way: use the Flownet2CSS weights provided by the Flownet code to generate DAVIS optical flow files?

But I'm still confused, it's only logical that the weights file you provided should be correct

[hzxie]

1. You can do that.
2. At the released time, I confirmed that the given link in the README is correct. Maybe the authors of FlowNet2 changed the code.

[theRain0450]

1. You can do that.

   2. At the released time, I confirmed that the given link in the README is correct. Maybe the authors of FlowNet2 changed the code.

Thank you very much for replying to my query in your busy schedule, now I completely understand!