Resnet in Python

[simra]

Hi, I took a crack at implementing the DeepCrossing residual layer in python and I'm interested in your feedback. I'd be happy to submit this code as a PR for addition to nn.py or as an additional feed-forward network example. One thing I'm uncertain about is what dimensionality to choose for the residual layers- currently I'm setting it to match the dimensionality of the initial dense layer.

def dense_layer(input, output_dim, nonlinearity):
    r = linear_layer(input, output_dim)
    if nonlinearity:
        r = nonlinearity(r)    
    return r;

def residual_layer(input, output_dim, inner_dim, nonlinearity):
    r=dense_layer(input,inner_dim,nonlinearity)
    r=dense_layer(r,output_dim,nonlinearity=None)
    r=plus(r,input)
    r=relu(r)
    return r

def fully_connected_classifier_resnet(input, num_output_classes, hidden_layer_dim, 
                                   num_hidden_layers, nonlinearity):

    h = dense_layer(input, hidden_layer_dim, nonlinearity)
    for i in range(1, num_hidden_layers):
        h = residual_layer(h, hidden_layer_dim, hidden_layer_dim, nonlinearity)
    r = linear_layer(h, num_output_classes)
    return r

[cha-zhang]

Could you follow the BrainScript implementation here: https://raw.githubusercontent.com/wiki/Microsoft/CNTK/Articles3/DeepCrossing.cntk

[eldakms]

Closing the issue due to inactivity. Feel free to reopen.