create an UpsampleLayer

[haixpham]

Hi delzac,

Do you have any idea how to create a layer wrapper for upsampling? I want to create a layer wrapper to use inside Sequential([]), but a naive implementation like this does not work:

def UpsamplingLayer(name=''):
    @C.BlockFunction('Upsampling', name=name)
    def inner(x):
        y = ops.upsample(x)
       return y
    return inner

This will raise exception because ops.upsample() calls reshape(), which does check x when the layer is created, but x is undefined at that time.

[delzac]

Unfortunately, based on the currently set of primitive op that is available, we definitely need to call shape. The alternative is for the user to provide the shape information (i.e. you have to pass the expected input shape in the arguments)

[haixpham]

But reshape() performs the check on x regardless of the shape we provide as argument. Which is a shame, would really be beautiful to have an upsampling layer. I really wanted to add more primitives in C++ but just didn't have the time.

Another question: what does @C.typemap mean when creating primitives?

[delzac]

Yup, so the shape information must be provided when reshape is called. The user will have to provide it when manually. :(

For C.typemap i'm not too sure. The docs weren't exactly clear on its meaning.

[delzac]

@haixpham I found a better way to upsample. Implemented it already. You can check it out here

[haixpham]

@delzac: lovely! Your upsampling function supports arbitrary scaling factor right?

I've been working with Pytorch lately in work project. Its implementation of RNN's batch processing is very awkward, make me appreciate CNTK's design more.

[delzac]

Yup, arbitrary integer scaling factor. The input must have shape (channel, spatial, spatial) though.

Interesting! I have been considering to shift to pytorch too, since there's more features and community support there. What's the learning curve like? How about speed and gpu memory usuage?

[haixpham]

Pytorch is not that different from Cntk, except for sequence handling: sequences in batch must be sorted before forward/backprop. I still misuse pad_pack and pack_pad to this day. Unlike Cntk, batch of sequence processing is simple.

Pytorch has the advantage of building computational graph for each data point, so it's more flexible. Speed is not that bad, given fast CPU the graph construction cost is negligible. It is still a bit slower than Cntk, though.

It's feature rich, but the downside is that the API is verbose, feel like Java.