PyTorch convert function for op reflection_pad2d not implemented. / What is context/node?

[mushipand]

❓Question

Hi! I got this error while converting GAN model(PyTorch) to mlmodel. As you said in https://developer.apple.com/videos/play/wwdc2020/10153/, I think I have to write reflection_pad2d by MIL.

My question is

• What does context and node means, which is the parameters used in the demo? Are they related to ONNX?
• Thanks to your update, we now do not have to convert pytorch model to ONNX. My understanding is that coremltools automatically do it instead of us. However, I found similar issue in ONNX's issue (https://github.com/onnx/onnx/issues/21) and according to this issue, reflection_pad2d (or the counterpart for ONNX) is now supported. Then, is my understanding wrong?

System Information

• torch == 1.5.0
• Core ML == 4.0b1 (I also tried 4.0b2)

Thank you!

[leovinus2001]

What does context and node means

Context is the set of converted operations as the converter goes through a structured dataflow graph of instructions. The node is the current instruction/operation being converted. Use the logging as discussed in issue #818 to print the model/graph during conversion. With that, you get insight in where in the model and graph a conversion fails.

[mushipand]

@leovinus2001 Thank you so much! I got those two concept at last! I'll try logging. Thank you for your help!

[3DTOPO]

I am having the same issue converting a PyTorch sequential model to CoreML. Even though the use of ONNX is no longer recommended, I was able to get a MLModel by converting from PyTorch to ONNX, then from ONNX to MLModel.

I would much rather go straight from PyTorch to MLModel, so if anyone has a solution, I would be grateful to learn about it.

[mushipand]

@3DTOPO Thank you! Actually, I also tried converting through ONNX, but I always failed in it. Could you tell me the version of onxx/onxx-coreml/coremltools? Some of the failure comes from ModuleNotFoundError.

[3DTOPO]

I struggled with it too.

I pip installed this:

onnx==1.7.0 coremltools==4.0b1 torch==1.5.1

Which gives me:

attr==0.3.1 attrs==19.3.0 future==0.18.2 mpmath==1.1.0 numpy==1.19.1 Pillow==7.2.0 protobuf==3.12.2 scipy==1.5.2 six==1.15.0 sympy==1.6.1 tqdm==4.48.0 typing-extensions==3.7.4.2

[mushipand]

Thanks a lot for quick response! I'll try as you did.

[mushipand]

@3DTOPO Hi! Actually, I could not have convert my model through ONNX, but thank you anyway!

So, I workaround the problem by implementing substitutive class for ReflectionPad2d(see below), and it seems to work for my network. (I also tried to write it in MIL, but it was hard for me) However, through conversion, Dimension mismatch occurred. Could you please try the new 'reflectionpad2d_rev' and see if it works fine to your network? Thanks in advance.

class ReflectPad2d_rev(nn.Module):
    '''
    new reflectionpad2d
        size : int (the size of padding)
    '''
    def __init__(self, size):
        super().__init__()
        self.size = size

    def forward(self, x):
        a = self.size
        L_list, R_list = [], []
        U_list, D_list = [], []
        for i in range(a):#i:0, 1
            l = x[:, :, :, (a-i):(a-i+1)]
            L_list.append(l)
            r = x[:, :, :, (i-a-1):(i-a)]
            R_list.append(r)
        L_list.append(x)
        x = torch.cat(L_list+R_list[::-1], dim=3)
        for i in range(a):
            u = x[:, :, (a-i):(a-i+1), :]
            U_list.append(u)
            d = x[:, :, (i-a-1):(i-a), :]
            D_list.append(d)
        U_list.append(x)
        x = torch.cat(U_list+D_list[::-1], dim=2)
        return x

I'm sorry but padding sizes are all identical in my case, so please revise it if you want to change the parameter to tuple. Because torch.flip is not supported either, the code is a bit messy.

[3DTOPO]

@mushipand wow, thank you so much! That does the trick!

I too wish I knew how to write MIL functions. Does anyone know of examples, tutorials or more information about them?

[leovinus2001]

You probably only need a composite operator. Maybe start here https://coremltools.readme.io/docs/composite-operators followed by https://coremltools.readme.io/docs/model-intermediate-language

Also see the PR https://github.com/apple/coremltools/pull/829/files for the composite-operators examples of sum() neg() topk()

And search for various examples for mv() and bmm() in the Issues of this repo.

[heydavid525]

Thanks for the discussion. We'll add reflection_pad2d in the coming releases.

[3DTOPO]

I was disappointed to find that reflection_pad2d is not supported with the 4.0 release. Now I will have to retrain my models.

[dearkafka]

I'm also hoping for reflection_pad2d support in future.

[vonholst]

CoreML has a similar op in the pad() operation. I think you can use this:

from coremltools.converters.mil import Builder as mb
from coremltools.converters.mil import register_torch_op
from coremltools.converters.mil.frontend.torch.ops import _get_inputs

@register_torch_op(torch_alias=["reflection_pad2d"])
def HackedReflection_pad2d(context, node):
    inputs = _get_inputs(context, node)
    x = inputs[0]
    pad = inputs[1].val
    x_pad = mb.pad(x=x, pad=[pad[2], pad[3], pad[0], pad[1]], mode='reflect')
    context.add(x_pad, node.name)

[eric573]

I would like to chime in to request for this feature as well!

[vonholst]

I have created a pull request for this. You can use that solution if you like.

[aseemw]

@vonholst 's PR is now part of coremltools 4.1