Open Rakshith2597 opened 1 year ago
@likholat, @alexeyhorkin, do you still have an alternative solution with MO Python extension which allows replace MaxUpooling to known ops? Cannot find in https://github.com/openvinotoolkit/openvino_notebooks/pull/487
Update: Never mind
@Rakshith2597, may I ask you to try this approach which let you don't use custom extensions at all: https://github.com/openvinotoolkit/openvino/pull/11400. You need create an alias for unpooling layer, export PyTorch model with it and then convert to OpenVINO IR using Model Optimized extension from the PR.
@dkurt , Here are the steps I did, please let me know if I am doing it right.
Got the following error:
OpenVINO runtime found in: /home/deeptensor/rakshith_codes/training_extensions/misc/pytorch_toolkit/lung_nodule_detection/venv/lib/python3.9/site-packages/openvino
OpenVINO runtime version: 2022.1.0-7019-cdb9bec7210-releases/2022/1
Model Optimizer version: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ ERROR ] Cannot infer shapes or values for node "ATen_53".
[ ERROR ] There is no registered "infer" function for node "ATen_53" with op = "ATen". Please implement this function in the extensions.
For more information please refer to Model Optimizer FAQ, question #37. (https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=37#question-37)
[ ERROR ]
[ ERROR ] It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ] Or because the node inputs have incorrect values/shapes.
[ ERROR ] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ] Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (<class 'openvino.tools.mo.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "ATen_53" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)
I think i might have missed something here
You need create an alias for unpooling layer, export PyTorch model with it
Could you please elaborate this? Sorry, I'm new to this.
Seems fine, but can you try operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK
?
I'm converting my PyTorch model to ONNX using,
torch.onnx.export(model, dummy_input, res_path,
input_names=['input'], output_names=['output'],
operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK,
verbose=False)
This raises the error mentioned in the previous comment.
@Rakshith2597, got it, thanks! Is there a way to share model definition so I can reproduce and propose a solution?
@dkurt , Sure, below is the model definition.
Additionally, I have provided the model definition, model weights, onnx file, error and debug log from mo in GDrive if you need it here. Thank you for helping. I really appreciate it.
class SUMNet(nn.Module):
def __init__(self,in_ch,out_ch):
super().__init__()
self.encoder = models.vgg11_bn(pretrained = True).features
self.preconv = nn.Conv2d(in_ch, 3, 1)
self.conv1 = self.encoder[0]
self.bn1 = self.encoder[1]
self.pool1 = nn.MaxPool2d(2, 2, return_indices = True)
self.conv2 = self.encoder[4]
self.bn2 = self.encoder[5]
self.pool2 = nn.MaxPool2d(2, 2, return_indices = True)
self.conv3a = self.encoder[8]
self.bn3 = self.encoder[9]
self.conv3b = self.encoder[11]
self.bn4 = self.encoder[12]
self.pool3 = nn.MaxPool2d(2, 2, return_indices = True)
self.conv4a = self.encoder[15]
self.bn5 = self.encoder[16]
self.conv4b = self.encoder[18]
self.bn6 = self.encoder[19]
self.pool4 = nn.MaxPool2d(2, 2, return_indices = True)
self.conv5a = self.encoder[22]
self.bn7 = self.encoder[23]
self.conv5b = self.encoder[25]
self.bn8 = self.encoder[26]
self.pool5 = nn.MaxPool2d(2, 2, return_indices = True)
self.unpool5 = nn.MaxUnpool2d(2, 2)
self.donv5b = nn.Conv2d(1024, 512, 3, padding = 1)
self.donv5a = nn.Conv2d(512, 512, 3, padding = 1)
self.unpool4 = nn.MaxUnpool2d(2, 2)
self.donv4b = nn.Conv2d(1024, 512, 3, padding = 1)
self.donv4a = nn.Conv2d(512, 256, 3, padding = 1)
self.unpool3 = nn.MaxUnpool2d(2, 2)
self.donv3b = nn.Conv2d(512, 256, 3, padding = 1)
self.donv3a = nn.Conv2d(256,128, 3, padding = 1)
self.unpool2 = nn.MaxUnpool2d(2, 2)
self.donv2 = nn.Conv2d(256, 64, 3, padding = 1)
self.unpool1 = nn.MaxUnpool2d(2, 2)
self.donv1 = nn.Conv2d(128, 32, 3, padding = 1)
self.output = nn.Conv2d(32, out_ch, 1)
def forward(self, x):
preconv = F.relu(self.preconv(x), inplace = True)
conv1 = F.relu(self.bn1(self.conv1(preconv)), inplace = True)
pool1, idxs1 = self.pool1(conv1)
conv2 = F.relu(self.bn2(self.conv2(pool1)), inplace = True)
pool2, idxs2 = self.pool2(conv2)
conv3a = F.relu(self.bn3(self.conv3a(pool2)), inplace = True)
conv3b = F.relu(self.bn4(self.conv3b(conv3a)), inplace = True)
pool3, idxs3 = self.pool3(conv3b)
conv4a = F.relu(self.bn5(self.conv4a(pool3)), inplace = True)
conv4b = F.relu(self.bn6(self.conv4b(conv4a)), inplace = True)
pool4, idxs4 = self.pool4(conv4b)
conv5a = F.relu(self.bn7(self.conv5a(pool4)), inplace = True)
conv5b = F.relu(self.bn8(self.conv5b(conv5a)), inplace = True)
pool5, idxs5 = self.pool5(conv5b)
unpool5 = torch.cat([self.unpool5(pool5, idxs5), conv5b], 1)
donv5b = F.relu(self.donv5b(unpool5), inplace = True)
donv5a = F.relu(self.donv5a(donv5b), inplace = True)
unpool4 = torch.cat([self.unpool4(donv5a, idxs4), conv4b], 1)
donv4b = F.relu(self.donv4b(unpool4), inplace = True)
donv4a = F.relu(self.donv4a(donv4b), inplace = True)
unpool3 = torch.cat([self.unpool3(donv4a, idxs3), conv3b], 1)
donv3b = F.relu(self.donv3b(unpool3), inplace = True)
donv3a = F.relu(self.donv3a(donv3b))
unpool2 = torch.cat([self.unpool2(donv3a, idxs2), conv2], 1)
donv2 = F.relu(self.donv2(unpool2), inplace = True)
unpool1 = torch.cat([self.unpool1(donv2, idxs1), conv1], 1)
donv1 = F.relu(self.donv1(unpool1), inplace = True)
output = self.output(donv1)
return output
@Rakshith2597, seems like the following steps helped me convert a model:
self.unpool1 = nn.MaxUnpool2d(2, stride=2)
...
unpool1 = torch.cat([self.unpool1(donv2, idxs1), conv1], 1) # not .apply
pool5
and unpool5
(have not looked how to fix it). Required only for upool5
.
unpool5 = torch.cat([self.unpool5(pool5 + 1e-10, idxs5), conv5b], 1)
operator_export_type=torch.onnx.OperatorExportTypes.ONNX_FALLTHROUGH
--- a/mo_extensions/front/onnx/max_unpool2d_decomposition.py
+++ b/mo_extensions/front/onnx/max_unpool2d_decomposition.py
@@ -20,7 +20,7 @@ class MaxUnpoolFrontReplacer(FrontReplacementSubgraph):
nodes=[
("max_pool0", dict(op="MaxPool")),
("max_pool1", dict(op="MaxPool")),
- ("slice", dict(op="AttributedSlice")),
+ ("slice", dict(op="Slice")),
("sub", dict(op="Sub")),
("unpool", dict(op="max_unpool2d")),
],
@dkurt Thanks a lot for your help. I am able to convert the model to IR.
@dkurt Hi, on trying to do inference with the ONNX model generated, I'm getting the following error.
[ONNXRuntimeError] : 1 : FAIL : Load model from downloads/model_weights/stage1/lung_seg.onnx failed:Fatal error: max_unpool2d is not a registered function/op
Code used to do inference
net = onnxruntime.InferenceSession('model.onnx')
ort_inputs = {net.get_inputs()[0].name: to_numpy(inputs)}
net_out = net.run(None, ort_inputs)
Code used to create ONNX model
torch.onnx.export(model, dummy_input, res_path,
input_names=['input'], output_names=['output'],
operator_export_type=torch.onnx.OperatorExportTypes.ONNX_FALLTHROUGH,
verbose=False)
Am I missing something?
Not sure that ONNX Runtime can handle this layer. I've tested OpenVINO 2022.2 and inference was fine.
Hi @dkurt , Your solution to overcome a pytorch to IR model conversion issue here was really helpful, however we couldn't proceed with the PR due to dataset license issues. I am facing issue with model conversion in another model, this time it is due to maxunpooling layer in the decoder. I came across your really good extension. However, I am still not able to resolve the problem.
Receiving the following the error while converting model with maxunpool.
Environment Specs:
You can find the model definition, model weights, onnx file, error and debug log from
mo
here.