Implement Split

[kilroythethird]

Feature request Currently the Split layer isn't implemented. This is a bit tricky to implement myself as it has by definition multiple outputs and i have no real clue how to implement this in onnx2keras. Any hints (or even an implementation) would be most welcome.

Additional context I am trying to port s3fd from https://github.com/1adrianb/face-alignment It also required Pow and Sqrt which where easy to implement using K functions. Crude implementation for both (untested as of yet):

def generic_K_converter(func, node, params, layers, node_name):
    inputs = [ensure_tf_type(layers[x], layers[list(layers)[0]]) for x in node.input]
    import keras.backend as K
    func = getattr(K, func)
    def target_layer(args):
        if not isinstance(args, list):
            args = [args]
        return func(*args, **params)
    lambda_layer = keras.layers.Lambda(target_layer, name=node_name)
    layers[node_name] = lambda_layer(inputs)

layers.AVAILABLE_CONVERTERS["Pow"] = partial(generic_K_converter, "pow")
layers.AVAILABLE_CONVERTERS["Sqrt"] = partial(generic_K_converter, "sqrt")

[gmalivenko]

Hello @kilroythethird. I implemented Split but haven't tested yet. If you are in hurry, you can check master branch of onnx2keras and try to convert your model.

[kilroythethird]

Thanks for the implementation. Sadly i run into the same error as with my own impl. It looks to me like it is a problem with Split returning multiple outputs and Max can't find the second one.

...
704 22:22:20.407424 140475945899648 converter.py:112] ######
W0704 22:22:20.407557 140475945899648 converter.py:113] ...
W0704 22:22:20.407622 140475945899648 converter.py:114] Converting ONNX operation
W0704 22:22:20.407676 140475945899648 converter.py:115] type: Split
W0704 22:22:20.407732 140475945899648 converter.py:116] node_name: 151
W0704 22:22:20.407785 140475945899648 converter.py:117] node_params: {'axis': 1, 'split': [1, 1, 1, 1]}
W0704 22:22:20.407844 140475945899648 converter.py:118] ...
W0704 22:22:20.407908 140475945899648 converter.py:125] Check input 0 (name 139).
W0704 22:22:20.407973 140475945899648 converter.py:135] ... found all, continue
W0704 22:22:20.412601 140475945899648 converter.py:112] ######
W0704 22:22:20.412700 140475945899648 converter.py:113] ...
W0704 22:22:20.412782 140475945899648 converter.py:114] Converting ONNX operation
W0704 22:22:20.412855 140475945899648 converter.py:115] type: Max
W0704 22:22:20.412929 140475945899648 converter.py:116] node_name: 155
W0704 22:22:20.413002 140475945899648 converter.py:117] node_params: {}
W0704 22:22:20.413076 140475945899648 converter.py:118] ...
W0704 22:22:20.413151 140475945899648 converter.py:125] Check input 0 (name 151).
W0704 22:22:20.413222 140475945899648 converter.py:125] Check input 1 (name 152).
W0704 22:22:20.413291 140475945899648 converter.py:127] The input not found in layers / model inputs.
Traceback (most recent call last):
  File "conv.py", line 12, in <module>
    k_model = pytorch_to_keras(model, input_var, [(3, 640, 640,)], verbose=True, change_ordering=False) 
  File "/home/nope/Downloads/face-alignment/torch2onnx.py", line 160, in pytorch_to_keras
    verbose=verbose, change_ordering=change_ordering)
  File "/home/nope/Downloads/face-alignment/onnx2keras/converter.py", line 133, in onnx_to_keras
    raise AttributeError('Current node is not in weights / model inputs / layers.')
AttributeError: Current node is not in weights / model inputs / layers.

Should i move this issue to onnx2keras btw, or is it ok here ?

[gmalivenko]

I checked your model by myself: gist

You can just pull the latest version of the onnx2keras from pypi: pip install --upgrade onnx2keras

DEBUG:onnx2keras:Converting ONNX operation
DEBUG:onnx2keras:type: Max
DEBUG:onnx2keras:node_name: 155
DEBUG:onnx2keras:node_params: {}
DEBUG:onnx2keras:...
DEBUG:onnx2keras:Check if all inputs are available:
DEBUG:onnx2keras:Check input 0 (name 151).
DEBUG:onnx2keras:Check input 1 (name 152).
DEBUG:onnx2keras:... found all, continue
DEBUG:onnx2keras:######
DEBUG:onnx2keras:...
DEBUG:onnx2keras:Converting ONNX operation
DEBUG:onnx2keras:type: Max
DEBUG:onnx2keras:node_name: 156
DEBUG:onnx2keras:node_params: {}
DEBUG:onnx2keras:...
DEBUG:onnx2keras:Check if all inputs are available:
DEBUG:onnx2keras:Check input 0 (name 155).
DEBUG:onnx2keras:Check input 1 (name 153).
DEBUG:onnx2keras:... found all, continue
DEBUG:onnx2keras:######

[kilroythethird]

Thank you again. Partial success.

Tensor("139/add:0", shape=(?, 4, 64, 64), dtype=float32)
Traceback (most recent call last):
  File "foo.py", line 151, in <module>
    k_model = onnx_to_keras(onnx_model, ['input'])
  File "/home/nope/Downloads/face-alignment/onnx2keras/converter.py", line 146, in onnx_to_keras
    node_name
  File "/home/nope/Downloads/face-alignment/onnx2keras/operation_layers.py", line 197, in convert_split
    layers[node_name] = lambda_layer(input_0)
  File "/home/nope/.local/lib/python3.7/site-packages/keras/engine/base_layer.py", line 457, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/nope/.local/lib/python3.7/site-packages/keras/layers/core.py", line 687, in call
    return self.function(inputs, **arguments)
  File "/home/nope/Downloads/face-alignment/onnx2keras/operation_layers.py", line 194, in target_layer
    return tf.split(x, splits, axis=axis)[isplit]
  File "/home/nope/.local/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 1568, in split
    "to split. Argument provided: %s" % (num_or_size_splits,))
ValueError: Rank-0 tensors are not supported as the num_or_size_splits argument to split. Argument provided: 4

with your gist. But from now on i should be able to fix the rest myself. Will raise PR/Issue if i end up patching stuff. Great tool btw, thanks for sharing it.

[kilroythethird]

@nerox8664 Quick additions: I had to patch convert_split to

def convert_split(node, params, layers, node_name):
    """
    Convert Split layer
    :param node: current operation node
    :param params: operation attributes
    :param layers: available keras layers
    :param node_name: resulting layer name
    :return: None
    """
    import numpy as np
    # output is a <class 'google.protobuf.pyext._message.RepeatedScalarContainer'> which can't get jsonified
    # TODO: do in proper location
    params['_outputs'] = list(params['_outputs'])

    if len(node.input) != 1:
        assert AttributeError('More than 1 input for split layer.')

    input_0 = ensure_tf_type(layers[node.input[0]])
    def target_func(input, split=params['split'], axis=params.get("axis", 0)):
        cur = 0
        ret = []
        slices = [slice(None, None)] * len(input.shape)
        for s in split:
            slices[axis] = slice(cur, cur+s)
            ret.append(input[slices])
            cur += s
        return ret
    splits = keras.layers.Lambda(target_func, name="%s_split" % node_name)
    splits = splits(input_0)
    for i, _ in enumerate(params['split']):
        node_name = params['_outputs'][i]
        layers[node_name] = splits[i]

and it works really well now. Faces are properly detected :)

[kilroythethird]

Sorry for mixing this all in one issue, but i am not sure if it is related to my above split implementation or not. When loading the model after saving it inclusive weights to an h5 file ensure_tf_type->target_layer receives floats and lists for the inp parameter. And as that function then tries to access the dtype of inp this leads to a crash. Checking if it is an instance of ndarray and otherwise skipping the dtype parameter works nicely. I am just not sure why that is happening at all.