Neural Network design

[MarcoROG]

We should discuss how the user will build a neural network in Syft and how it will be implemented internally.

Do we want each Layer (Dense, Convolution..) to be its own class containing weights and all the information? (Unlike the linear example we currently have).

Do we want a "Model" wrapper containing meta-information?

Overall we should discuss how we want to design this structure and carry out the development.

[hohmannr]

Alright, have been waiting fro this kinda :D! In my opinion, we have to give the user/data scientist a lot of flexibility. They should be able to define their model through a simple step process by:

1. defining the input tensor
2. building the NN's hidden layers, layer after layer
3. defining activation funcs for each seperate layer independently (2+3 can be done by using a layer class for example)
4. creating an own training loop and
5. defining their backpropagation algorithm

Since we have to deal with a wide variety of NNs, I would say we start with classical ANNs and build on top of them. Also I would consider pytorchs or tensorflows syntax since they are used widely and if we implement a similiar fasion, the learning process for the data scientist wouldn't be that steep.

[MarcoROG]

Also consider we have Autograd functionality planned, so that should be taken in consideration as well, the internal implementation must be "usable" for that as well.

What you proposed sounds a lot like Keras, am I right? The idea is to have everything as a layer. I would not tie layers to have an activation function, as sometimes this is not the case (there are some papers where 2 convolutions without an activation inbetween are used as a form of convolution instead of using a single, bigger filter size)

[hohmannr]

Yes, what I meant was enabling Modifications for each layer (of what kind ever: convolution, activation funcs, pooling, flattening etc.) The best thing in my opinion would be a class NN, that the data scientist can inherit from, which implements GPU support and Encryption on its own and the data scientist has to implement some default methods (e.g. train_cycle() or build()) by himself and can modify the class however he would like to.

[MarcoROG]

This could require a lot of coding and most of the times it won't be needed. I think we should err on the side of "make the simple case simple and the corner case possible". We could provide the possibility to inherit from it but I think that would be an extra. 99% of the usage should be doable just by using the existing class as a client. Thats what makes keras so easy to use

[MarcoROG]

https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py As a reference : I think 99% of the use cases should be as simple as this

[hohmannr]

Well, I think that this implementation of the Sequential() class could cover our base NN class, but since most model development simply needs more depth and tuning, I would also like to be able to inherit from this class and e.g. write my own activation funcs or convolution filters, so I can exactly fit the model to my problem and my specific needs (im really OOP friendly). Another solution would be to fit the model to ones needs through using custom created layer classes. But there we would also need a guideline on what the data sc. needs to implement in order for them to work.

In general, I think that a simple class with base functionality is not avoidable. How we implement use case specifics and custom development we'll have to discuss, because there are a wide variety of options to consider.

[MarcoROG]

I would opt for custom layer classes if you Wang customization. We could also look at pytorch implementation, which is less lego-like and more similar to what you said

[hohmannr]

Yea, that's also what Im using for my models >:D, at the end it comes down to preference.

[hohmannr]

Lets get some other opinions in here :D

[cosmadrian]

I think since syft is inspired by PyTorch, that we should kind of mimic that interface, having enough abstractions but not that much magic behind it.

[siarez]

Different levels of abstraction are available through PyTorch's API. That is basic/low-level functions are implemented in the torch.nn.functional module. At a higher level, torch.nn layers make use of the methods in the torch.nn.functional module and inherit from torch.nn.Module. And at a yet higher level, you can use different containers(which also inherit from torch.nn.Module) to stack and wrap your torch.nn layers. Or you can define your own container and layers by inheriting from torch.nn.Module I haven't used Keras, but looks very similar to Keras, and also grant everyones wishes, no?

[bharathgs]

1. I totally agree with @siarez and @cosmadrian we should stick with pytorch-esque design. (Ref: torch.nn & torch.nn.functional)
2. However, If users want something more keras-esque they can always have something similar to aorun on top of pysyft. But we should focus on the prior.

[hohmannr]

Totally agree with my three pre-commentators! Implement the Basic stuff first and then we can write a wrapper around it on top!