DNN Torch import is missing several layers

[cyberbeat]

System information (version)

• OpenCV => master (4.2.x)
• Operating System / Platform => Linux 64 Bit
• Compiler => gcc9

Detailed description

I try to import a dnn model from torch, and several layers cannot be created.

The model is for deblurring images: https://arxiv.org/abs/1612.02177

This is the code: https://github.com/SeungjunNah/DeepDeblur_release with links to the model-files: https://drive.google.com/file/d/1Z8dV6KuubfOKj4ganEjxymhyMoXoydfo/view?usp=sharing

The missing torch layers are:

• PixelShuffle
• Copy
• AddConstant
• ParallelTable
• FlattenTable
• NarrowTable
• SelectTable

I think I got Copy and AddConstant simply implemented via custom layer (remap to opencv layers). I also tried PixelShuffle, I am not sure, if I got it right. But the table layers seem to be difficult with custom layers in opencv?

[czgdp1807]

I would like to take up this issue.

[czgdp1807]

@cyberbeat Can you please share the CPP/Python code which you used to import the .t7 files. I tried to find ParallelTable, FlattenTable, SelectTable in PyTorch but they didn't appear to be there.

>>> dir(torch.nn.FlattenTable)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch.nn' has no attribute 'FlattenTable'
>>> dir(torch.nn.SelectTable)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch.nn' has no attribute 'SelectTable'
>>> dir(torch.nn.PixelShuffle)
['__call__', '__class__', '__constants__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_apply', '_get_name', '_load_from_state_dict', '_named_members', '_register_load_state_dict_pre_hook', '_register_state_dict_hook', '_replicate_for_data_parallel', '_save_to_state_dict', '_slow_forward', '_version', 'add_module', 'apply', 'buffers', 'children', 'cpu', 'cuda', 'double', 'dump_patches', 'eval', 'extra_repr', 'float', 'forward', 'half', 'load_state_dict', 'modules', 'named_buffers', 'named_children', 'named_modules', 'named_parameters', 'parameters', 'register_backward_hook', 'register_buffer', 'register_forward_hook', 'register_forward_pre_hook', 'register_parameter', 'requires_grad_', 'share_memory', 'state_dict', 'to', 'train', 'type', 'zero_grad']

[cyberbeat]

Torch 7 Manual here lists the layers:

https://nn.readthedocs.io/en/rtd/table/index.html

The torch source code for the layers is also available here:

https://github.com/torch/nn/blob/master/ParallelTable.lua

This was my code for opencv:

int main() { Net net = readNetFromTorch("DeepDeblur_release/experiment/experiment-cpu/release_scale3_adv_gamma/models/model-450.t7"); Mat im = imread("DeepDeblur_release/images/Istanbul_blur1.png"); Mat blob = blobFromImage(im); net.setInput(blob); Mat out; net.forward(out); return 0; }

[czgdp1807]

Thank you. I will try to reproduce the bug/absence of layers and if observed I will try to fix it.

[czgdp1807]

Hi, the model is being read perfectly. Can you please comment the error message which you got? I didn't get any related to readNetFromTorch.

[cyberbeat]

Did you also did net.forward(out)? There the errors appear there, if I remember right.

When you look at the opencv code here: opencv/modules/dnn/src/torch/torch_importer.cpp you'll see, that many layers are missing.

[czgdp1807]

I got the following exception. Please let me know if it was something similar in your case. I will try to come up with the APIs of the new layers. It would be great if you can share your code which you wrote for Copy, AddConstant, and PixelShuffle layers.

terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(3.4.9-dev) /home/czgdp1807ssd/opencv_project/opencv/modules/dnn/src/dnn.cpp:565: error: (-2:Unspecified error) Can't create layer "l1_Copy" of type "Copy" in function 'getLayerInstance'

[czgdp1807]

I have though of the following API for SelectTable layer,

There will be a class SelectTable inheriting Layer. Data Members

1. index - An integer specifying the index of the element which is to be output.
2. gradInput - An array of Matrices representing the table.
3. output - The output of the layer.
4. gradOutput

Class methods

1. updateOutput(input) - This will update the output data member according to the input received.
2. updateGradInput(input, gradOutput)

References .. [1] https://github.com/torch/nn/blob/master/SelectTable.lua .. [2] https://nn.readthedocs.io/en/rtd/table/index.html#nn.SelectTable

Please let me know what you think. ping @alalek @cyberbeat

[cyberbeat]

I am not sure, if I have done it right, but here is my code:

class PixelShuffleLayer : public cv::dnn::Layer
{
public:
    PixelShuffleLayer(const cv::dnn::LayerParams ¶ms){
        setParamsFrom(params);
        upscaleFactor = params.get("upscaleFactor");
    }
    static cv::Ptr create(cv::dnn::LayerParams& params) {
        return cv::Ptr(new PixelShuffleLayer(params));
    }
    virtual bool getMemoryShapes(const std::vector &inputs,
                                 const int requiredOutputs,
                                 std::vector &outputs,
                                 std::vector &internals) const CV_OVERRIDE {

        outputs = std::vector(inputs.size(), shape(
                   inputs[0][0],
                   inputs[0][1] / (upscaleFactor * upscaleFactor),
                   inputs[0][2] * upscaleFactor,
                   inputs[0][3] * upscaleFactor));
        return false;
    }

    virtual void forward(InputArrayOfArrays inputs_arr, OutputArrayOfArrays outputs_arr, OutputArrayOfArrays internals) CV_OVERRIDE {
        std::vector inputs, outputs;
        inputs_arr.getMatVector(inputs);
        outputs_arr.getMatVector(outputs);

        Mat inp = inputs[0];
        Mat out = outputs[0];

        permuteLayer->forward(inp, out, internals);
        vector outputShape { inp.size[0], inp.size[1] / (upscaleFactor * upscaleFactor),  inp.size[2] * upscaleFactor, inp.size[3] * upscaleFactor};
        out.reshape(1, outputShape);
    }

    virtual void finalize(cv::InputArrayOfArrays inputs_arr,
                          cv::OutputArrayOfArrays outputs_arr) CV_OVERRIDE {
        std::vector inputs, outputs;
        inputs_arr.getMatVector(inputs);
        outputs_arr.getMatVector(outputs);

        Mat inp = inputs[0];
        Mat out = outputs[0];

        LayerParams permuteLayerparams;
        int order[] = { 0, 3, 1, 4, 2};
        permuteLayerparams.set("order", DictValue::arrayInt(&order[0], 5));
        vector permuteOutShape, permuteInpShape { inp.size[0], inp.size[1] / upscaleFactor, inp.size[1] / upscaleFactor, inp.size[2] * upscaleFactor, inp.size[3] * upscaleFactor};
        permuteOutShape.resize(5);
        for (int i = 0; i < 5; ++i)
            permuteOutShape[i] = permuteInpShape[order[i]];
        Ptr permuteLayer = PermuteLayer::create(permuteLayerparams);
        std::vector permuteInputs(1, inp.reshape(1, permuteInpShape));
        std::vector permuteOutputs(1, out.reshape(1, vector{ inp.size[0], inp.size[1] / (upscaleFactor * upscaleFactor),  inp.size[2] * upscaleFactor, inp.size[3] * upscaleFactor}));

    }

private:
    int upscaleFactor;
    Ptr permuteLayer;
};

static Ptr createAddConstantLayer(LayerParams& params){

    LayerParams p;
    p.set("shift", params.get("constant_scalar"));
    return PowerLayer::create(p);
}

And then

CV_DNN_REGISTER_LAYER_CLASS(Copy, BlankLayer);
    CV_DNN_REGISTER_LAYER_CLASS(PixelShuffle, PixelShuffleLayer);
    CV_DNN_REGISTER_LAYER_FUNC(AddConstant, createAddConstantLayer);

[czgdp1807]

Thank you. I will try to make a PR for this issue by tomorrow.

[cyberbeat]

@czgdp1807 Did you give up this task?