Yet another approach to implement Net module with numpy for vanilla neural network

[yiakwy]

Strengthen your network layers operator implementation

Hi, I just found this repo and I found that the network implementation is weak. Here I provide your some examples where you expand neural network implementation with tests covered:

Detailed explanation with Conv, BatchNorm, Relu, FullyConnected, UpSampling: https://github.com/yiakwy/yiakwy.github.io/blob/master/Correlation%20Metrics/ConvNet/costImpl/ConvNet/layers.py

Support keras alike layers stacking syntax:

Unit test covered: https://github.com/yiakwy/yiakwy.github.io/blob/master/Correlation%20Metrics/ConvNet/READMME.md

This implementation is aimed to give people the best understand of some operators like Upsampling.

Nowadays, many people has already understand how to implement operators in CPU devices. We call it vanilla implementation. It is still challenging to understand how to implement and optimize them in different devices.

When we talk about Neural Network as a Computing Graph, we are actually interested in how to implement operations planned in different devices. At least, I need to provide FLops needed in each operator as edge weight and understand how to implement them and distribute them in different devices.

[ddbourgin]

Hi @yiakwy - nice work on your repo! At this stage I don't have plans to implement things like an autograd system / computational graph functionality. If you're looking for something like that, see the amazing autograd repo.

I'd love to support syntax similar to the Keras functional API in the future, but it's not there yet. One more item on the TODO list :)

[yiakwy]

@ddbourgin Please check my latest update on Upsampling where I demonstrated that Keras employed a weak version bilinear upsampling (just like unpooling), and dilated convolution and bilinear convolution achieves the best performance by not only approximate overall features but also reserves details in upsampled feature maps.

Original:

Upsampled by Keras equivalent unpooling and coarse bilinear interpolation method by repeating downsampled feature maps.

Dilated Convolution and Bilinear Interpolation with Maximum Sampling Rate (see Shanno theorem for details)

I will show you simple use tranposed convolution does not work: