Resnet

[FrancescoSaverioZuppichini]

Implement ResNet in a scalable way

TODO

• [x] Draw the model in Figma with every sublayer
• [x] Implement the model in a scalable way
• [x] Trace and store the model weights using torchvision weights
• [x] add preactivation module from Identity Mappings in Deep Residual Networks

[francescocicala]

In nn.resnet we could subclass ResNetBottleNeckBlock and ResNetBasicBlock from a parent class by overwriting self.block, since they differ only for that attribute.

[FrancescoSaverioZuppichini]

Probably the best thing is to have a class ResidualBlock

class ResidualAddBlock(nn.Module):
    def __init__()
        super().__init__()
        self.block = None
        self.shortcut = None

    def forward(self, x: Tensor) -> Tensor:
        res = x
        if self.shortcut is not None:
            res = self.shortcut(res)
        x = self.block(x)
        x += res
        return x

class ResNetBasicBlock(ResidualAddBlock):
   def __init__(...):
      self.block = // weights here
      self.shortcut = ResNetShortcut if self.shold_apply_shortcut ....
      self.act = // relu
   def forward(self, x):
       x = super()(x)
       return self.act(x)

class ResNetBottleNeckBlock(ResNetBasicBlock):
   def __init__(...):
      self.block = // bottleneck weights here

[FrancescoSaverioZuppichini]

We need to write a test for ResNet, I think a good lazy idea can be just passing input to the model. If we have an output it means that at least the weights features are correct. What do you think?

[FrancescoSaverioZuppichini]

Add parameter shortcut to ResNetBasicBlock

[FrancescoSaverioZuppichini]

Done!

[FrancescoSaverioZuppichini]

Add wide resnet from "Wide Residual Networks" and resnetxt "Aggregated Residual Transformation for Deep Neural Networks

[FrancescoSaverioZuppichini]

Add Bag of Tricks for Image Classification with Convolutional Neural Networks