AlexNet bacward shape missmatch + ReLu return a tuple

[Belegkarnil]

Hi,

I have implemented AlexNet in singa but I obtain an error during the backward_and_update instruction. I am using Singa 3.0.0.rc1 on cpu.

This is my AlexNet implementation: `from singa import autograd from singa import module from singa import opt

all = ['AlexNet', 'alexnet']

class AlexNet(module.Module): def init(self, num_classes=1000): super(AlexNet, self).init()

12 sur GPU donc 6 & 6

    self.features1 = [
        autograd.Conv2d(3,64,kernel_size=11,stride=4,padding=2),
        autograd.ReLU(),
        autograd.MaxPool2d(kernel_size=3, stride=2),
        autograd.Conv2d(64,192,kernel_size=5,padding=2),
        autograd.ReLU(),
        autograd.MaxPool2d(kernel_size=3, stride=2),
        autograd.Conv2d(192,384,kernel_size=3,padding=1),
        autograd.ReLU(),
        autograd.Conv2d(384, 256,kernel_size=3,padding=1),
        autograd.ReLU()
    ]
    self.features2 = [
        autograd.Conv2d(256, 256,kernel_size=3,padding=1),
        autograd.ReLU(),
        autograd.MaxPool2d(kernel_size=3, stride=2)
    ]
    self.avgpool = autograd.AvgPool2d(6, stride=1)
    self.flatten = autograd.Flatten()
    self.classifier = [
        autograd.Dropout(),
        autograd.Linear(256 * 6 * 6, 4096),
        autograd.ReLU(),
        autograd.Dropout(),
        autograd.Linear(4096, 4096),
        autograd.ReLU(),
        autograd.Linear(4096, num_classes)
    ]
    self.optimizer = opt.SGD(lr=0.001, momentum=0.9)
def loss(self, out, ty):
    return autograd.softmax_cross_entropy(out, ty)
def optim(self, loss, dist_option, spars):
    if dist_option == 'fp32':
        self.optimizer.backward_and_update(loss)
    elif dist_option == 'fp16':
        self.optimizer.backward_and_update_half(loss)
    elif dist_option == 'partialUpdate':
        self.optimizer.backward_and_partial_update(loss)
    elif dist_option == 'sparseTopK':
        self.optimizer.backward_and_sparse_update(loss, topK=True, spars=spars)
    elif dist_option == 'sparseThreshold':
        self.optimizer.backward_and_sparse_update(loss, topK=False, spars=spars)
def forward(self, x):
    for (i,layers) in enumerate([self.features1, self.features2, [ self.avgpool,self.flatten  ] , self.classifier]):
        for (j,fn) in enumerate(layers):
            x = fn(x)
            if(type(x) is tuple):# FIXME I have to do that because of a bug in Singa? (ReLU)
                x = x[0]
    return x

def alexnet(kwargs): return AlexNet(kwargs) ` And I get : AssertionError: ('shape mismatch', (9216, 4096), (256, 4096)) Which is my first linear layer : 256 6 6, 4096

When I use my VGG16 implementation, I got a similar error : AssertionError: ('shape mismatch', (25088, 4096), (512, 4096))

It seems that the backward operation does not map the correct shape to the corresponding layer.

Moreover, the ReLu class return a 1-tuple containing a Tensor. Is it intended or is it a bug?

[dcslin]

Hi, as pointed out by @chrishkchris , the convention is to use RELU as stateless layer. usage: https://github.com/apache/singa/blob/master/examples/cnn/model/cnn.py#L40

For shape mismatch, you might need to check the shape of layers again. Let me know if further info is required.

[Belegkarnil]

Ok, I'll try but why to provide a statefull ReLU Layer? Is it for a specific purpose?

[Belegkarnil]

I compared my implementation to other frameworks and it is the same shapes. Moreover the forward pass does not cause any issue, it is the backward pass. This is why I suspect a bug. Is it possible?

[nudles]

Hi, as pointed out by @chrishkchris , the convention is to use RELU as stateless layer. usage: https://github.com/apache/singa/blob/master/examples/cnn/model/cnn.py#L40

For shape mismatch, you might need to check the shape of layers again. Let me know if further info is required.

@dcslin Did you try to run the code pasted by @Belegkarnil ? Can you reproduce the error?

[dcslin]

Hi, as pointed out by @chrishkchris , the convention is to use RELU as stateless layer. usage: https://github.com/apache/singa/blob/master/examples/cnn/model/cnn.py#L40 For shape mismatch, you might need to check the shape of layers again. Let me know if further info is required.

@dcslin Did you try to run the code pasted by @Belegkarnil ? Can you reproduce the error?

I am still checking the code

[dcslin]

Hi @Belegkarnil, you might need to change 256 6 6, 4096 to 256, 4096 to make it works.

Also you are recommended to use relu/dropout/flatten like this https://github.com/apache/singa/blob/master/examples/cnn/model/cnn.py#L40

[Belegkarnil]

Ok thanks a lot ! I assumed that it works like other frameworks but that the result of AvgPool has a different shape.