Error when converting Pytorch model to IR

[Miranda0920]

Platform (like ubuntu 16.04/win10):

ubuntu 16.04

Python version:

python 3.5.2

Source framework with version (like Tensorflow 1.4.1 with GPU):

Pytorch 0.4.0 with GPU

Destination framework with version (like CNTK 2.3 with GPU):

caffe

Pre-trained model path (webpath or webdisk path):

https://github.com/DagnyT/hardnet/tree/master/pretrained/train_liberty_with_aug

Running scripts:

import sys
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.backends.cudnn as cudnn
import time
import os
import cv2
import math
import numpy as np
class L2Norm(nn.Module):
    def __init__(self):
        super(L2Norm,self).__init__()
        self.eps = 1e-10
    def forward(self, x):
        norm = torch.sqrt(torch.sum(x * x, dim = 1) + self.eps)
        x= x / norm.unsqueeze(-1).expand_as(x)
        return x
class L1Norm(nn.Module):
    def __init__(self):
        super(L1Norm,self).__init__()
        self.eps = 1e-10
    def forward(self, x):
        norm = torch.sum(torch.abs(x), dim = 1) + self.eps
        x= x / norm.expand_as(x)
        return x
class HardNet(nn.Module):
    """HardNet model definition
    """
    def __init__(self):
        super(HardNet, self).__init__()

self.features = nn.Sequential(
           nn.Conv2d(1, 32, kernel_size=3, padding=1, bias = False),
           nn.BatchNorm2d(32, affine=False),
           nn.ReLU(),
           nn.Conv2d(32, 32, kernel_size=3, padding=1, bias = False),
           nn.BatchNorm2d(32, affine=False),
           nn.ReLU(),
           nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1, bias = False),
           nn.BatchNorm2d(64, affine=False),
           nn.ReLU(),
           nn.Conv2d(64, 64, kernel_size=3, padding=1, bias = False),
           nn.BatchNorm2d(64, affine=False),
           nn.ReLU(),
           nn.Conv2d(64, 128, kernel_size=3, stride=2,padding=1, bias = False),
           nn.BatchNorm2d(128, affine=False),
           nn.ReLU(),
           nn.Conv2d(128, 128, kernel_size=3, padding=1, bias = False),
           nn.BatchNorm2d(128, affine=False),
           nn.ReLU(),
           nn.Dropout(0.1),
           nn.Conv2d(128, 128, kernel_size=8, bias = False),
           nn.BatchNorm2d(128, affine=False),
        )
    def input_norm(self,x):
        flat = x.view(x.size(0), -1)
        mp = torch.mean(flat, dim=1)
        sp = torch.std(flat, dim=1) + 1e-7
        return (x - mp.detach().unsqueeze(-1).unsqueeze(-1).unsqueeze(-1).expand_as(x)) / sp.detach().unsqueeze(-1).unsqueeze(-1).unsqueeze(1).expand_as(x)
    def forward(self, input):
        x_features = self.features(self.input_norm(input))
        x = x_features.view(x_features.size(0), -1)
        return L2Norm()(x)
if __name__ == '__main__':
    DO_CUDA = True
    try:
          input_img_fname = sys.argv[1]
          output_fname = sys.argv[2]
          if len(sys.argv) > 3:
              DO_CUDA = sys.argv[3] != 'cpu'
    except:
          print("Wrong input format. Try ./extract_hardnet_desc_from_hpatches_file.py imgs/ref.png out.txt gpu")
          sys.exit(1)
    model_weights = '../pretrained/train_liberty_with_aug/checkpoint_liberty_with_aug.pth'
    model = HardNet()
    checkpoint = torch.load(model_weights)
    model.load_state_dict(checkpoint['state_dict'])
    from mmdnn.conversion.pytorch.pytorch_parser import PytorchParser
    size=32
    pytorchparser = PytorchParser(model, [1, size, size])
    IR_file = 'HardNet'
    pytorchparser.run(IR_file)

When i run the code , error occurs:

/usr/local/lib/python3.5/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Traceback (most recent call last): File "./mmdnn_ori_extract_hardnet_desc_from_hpatches_file.py", line 105, in pytorchparser = PytorchParser(model, [1, size, size]) File "/home/kangrong/envs/mmdnntest/lib/python3.5/site-packages/mmdnn/conversion/pytorch/pytorch_parser.py", line 67, in init if not os.path.exists(model_file_name): File "/home/kangrong/envs/mmdnntest/lib/python3.5/genericpath.py", line 19, in exists os.stat(path) TypeError: argument should be string, bytes or integer, not HardNet

I want to convert Pytorch model to caffe , but when i convert Pytorch to IR errors occur. How can i solve this error? And how can i convert Pytorch model to caffe model?

[namizzz]

Hi @kr0920 ,as you can see from the error log, the first argument should be model_file_name not a model object.You need to save your HardNet as a model file, then feed the model_file_name to PytorchParser. Thanks!

[Miranda0920]

@namizzz Thank you for your reply. I just fixed the problem using the method mentioned in the https://github.com/Microsoft/MMdnn/issues/241

Commit pytorch_parser.py from Line 67 to Line 76, and add model = model_file_name at Line 77

[Miranda0920]

Hi @namizzz , I am trying to convert Pytorch model to caffe . I meet some errors and i have no idea how to solve them.
1、I follow the instruction https://github.com/Microsoft/MMdnn/tree/master/mmdnn/conversion/pytorch to convert my pytorch model. I save my model using torch.save(model,path) ,so i think i do not need to use mmdownload to extract pre-trained model to standard model. Am i right?
So i directly run the command "mmtoir -f pytorch -d HardNet --inputShape 1 32 32 -n ../HardNet/hardnet/code/data/models/tfeat_whole/liberty_train_tfeat_whole/_liberty_min_as_fliprot/checkpoint_0.pth" to convert pytorch model to IR where 'HardNet' is my model name. But error occurs as below:

Traceback (most recent call last): File "/MMdnn/mmdnn/conversion/pytorch/pytorch_parser.py", line 74, in init model = torch.load(model_file_name) File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 303, in load return _load(f, map_location, pickle_module) File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 469, in _load result = unpickler.load() AttributeError: Can't get attribute 'HardNet' on <module 'main' from '/envs/mmdnn/bin/mmtoir'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/envs/mmdnn/bin/mmtoir", line 11, in load_entry_point('mmdnn', 'console_scripts', 'mmtoir')() File "/MMdnn/mmdnn/conversion/_script/convertToIR.py", line 187, in _main ret = _convert(args) File "/MMdnn/mmdnn/conversion/_script/convertToIR.py", line 82, in _convert parser = PytorchParser(args.network, inputshape) File "/MMdnn/mmdnn/conversion/pytorch/pytorch_parser.py", line 76, in init model = torch.load(model_file_name, map_location='cpu') File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 303, in load return _load(f, map_location, pickle_module) File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 469, in _load result = unpickler.load() AttributeError: Can't get attribute 'HardNet' on <module 'main' from '/envs/mmdnn/bin/mmtoir'>

２、Then I try to follow the tutorial in https://github.com/Microsoft/MMdnn/issues/241 . I use " torch.save(model,path) " to save the whole model and then I modify my conversion script as below

model_weights = '/my/path/to/.pth'
model = torch.load(model_weights)
from mmdnn.conversion.pytorch.pytorch_parser import PytorchParser
size=32
pytorchparser = PytorchParser(model, [1, size, size])
IR_file = 'HardNet'
pytorchparser.run(IR_file)

But there are still errors:

Traceback (most recent call last): File "mmdnn_extract_hardnet_desc_from_hpatches_file.py", line 106, in pytorchparser = PytorchParser(model, [1, size, size]) File "/MMdnn/mmdnn/conversion/pytorch/pytorch_parser.py", line 83, in init self.pytorch_graph.build(self.input_shape) File "/MMdnn/mmdnn/conversion/pytorch/pytorch_graph.py", line 143, in build output_shape = [int(x) for x in output_shape_str.split(',')] File "/MMdnn/mmdnn/conversion/pytorch/pytorch_graph.py", line 143, in output_shape = [int(x) for x in output_shape_str.split(',')] ValueError: invalid literal for int() with base 10: ' 1!'

3、How can i solve these problems and how can i convert my pre-trained Pytorch model to caffe ?

Looking forward to your reply.

[namizzz]

Hi @kr0920 , Thanks for your issue. I find there are some node like:

%27 : Dynamic = prim::Undefined(), scope: HardNet/Sequential[features]/BatchNorm2d[1]
%28 : Dynamic = prim::Undefined(), scope: HardNet/Sequential[features]/BatchNorm2d[1]

MMdnn pytorch parser can't support dynamic node now.

[ys198918]

hi @namizzz ,what does dynamic node mean?

[namizzz]

Hi @ys198918 ,from this :

dynamic means that the shape is not known. This may be because of a limitation of our implementation (which we would like to fix in a future release) or shapes which are truly dynamic.

[bowenc0221]

@kr0920 I encountered the same problem when trying to convert a customized PyTorch model. Have you solved this problem?

[ShuaiHuang]

Hi @kr0920 , Thanks for your issue. I find there are some node like:

%27 : Dynamic = prim::Undefined(), scope: HardNet/Sequential[features]/BatchNorm2d[1]
%28 : Dynamic = prim::Undefined(), scope: HardNet/Sequential[features]/BatchNorm2d[1]

MMdnn pytorch parser can't support dynamic node now.

Hi @namizzz , I encountered the same issue, and the dynamic node is pasted as following

%312 : Dynamic = onnx::Constant[value=   1  256    1    1 [ CPULongTensor{4} ]](), scope: FastPose_SE/SEResnet[preact]/Sequential[layer1]/Bottleneck_With_Reduction_64_64[0]/SELayer_256[se]

and the source code is

class SELayer(nn.Module):
    def __init__(self, channel):
        super(SELayer, self).__init__()
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Sequential(
            nn.Linear(channel, channel),
            nn.ReLU(inplace=True),
            nn.Linear(channel, channel),
            nn.Sigmoid()
        )

    def forward(self, x):
        b, c, _, _ = x.size()
        y = self.avg_pool(x).view(b, c)
        y = self.fc(y).view(b, c, 1, 1)
        return x * y

I tried to change the dynamic node to non-dynamic node as follwoing, but it seems doesn't work.

class SELayer_256(nn.Module):
    def __init__(self):
        super(SELayer_256, self).__init__()
        self.avg_pool = nn.AvgPool2d([80, 64])
        self.fc = nn.Sequential(
            nn.Linear(256, 256),
            nn.ReLU(inplace=True),
            nn.Linear(256, 256),
            nn.Sigmoid()
        )

    def forward(self, x):
        y = self.avg_pool(x).view(1, 256)
        y = self.fc(y).view(1, 256, 1, 1)
        return x * y

Could you please to give me some advice? Thank you in advance.

[konioyxgq]

@ShuaiHuang I encountered the same issue. Did you solve it, please?

[ShuaiHuang]

@ShuaiHuang I encountered the same issue. Did you solve it, please?

Nope. I tried the MACE and the MNN frameworks, and both of them works well.