Deep Neural Network initial implementation

[osanseviero]

As mentioned in #7, we would like to have a basic implementation of a Deep Neural Network. The design should be consistent with other major libraries and should have enough flexibility to add other types of layers (for the future when we add things like RNN or CNN).

Here is an example with Keras

model = Sequential()
model.add(Dense(512, input_dim=1000))
model.add(Dense(2, activation='softmax'))
model.compile(loss = 'mean_squared_error', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=200, batch_size=100, verbose=0)

And a complex one in TFLearn.

We need to define our interface as well.

So, what do we need?

• A model structure (TBD). Keras creates the model from the beginning and adds new layers to it. TFLearn connects the layers one by one, creating a net, and then runs tflearn.regression to the last layer. What do you think works better? How would you like you like to see users to create neural networks?
• Layer structure - this should be composed of nodes.
  • Implement a Node class. Nodes should have a value and lists for inbound and outbound nodes.
  • Implement an Input class (type of Node). This shouldn't have any inbound nodes.
  • Implement a Linear class (type of Node as well).
    • This one should take as input 3 things: values (matrix of values - to explain later), weights (list of weights), and a bias (just a number). Calculating the value of the linear node is done with this formula, where wi and xi correspond to the weights and values, and b is the bias. Numpy has some functions that should make this easy (np.dot).
  • Nodes should have a forward and backward method (I'll explain this later).
• Finally, we need to define forward and backward passes. In the forward pass, a value goes through the neural network. This is easy, as each node will have a forward method, so we will need to go through each node (To do this, we need to do a topological sort, which is the complex part. This will allow us to go through the graph in an ordered way). The backpropagation step is more complicated and requires some calculus, but, for now, just focus in implementing the forward pass.

@EdgarGar and @Biller17 what do you think?

[osanseviero]

A resource that should be helpful to understand the forward pass and how to implement the nodes. http://neuralnetworksanddeeplearning.com/chap1.html.

Note: in this tutorial, the author is adding a sigmoid activation to all the nodes. We shouldn't do this - doing np.dot(w, x)+b should be enough. Also, look at how he is setting up random initial weights. This is ok (probably, in the future, we might want to add some options around this). The weights are adjusted with backpropagation.