rnnlib

A library for (recurrent) neural networks in Dlang

The aim of this project is to build a RNN factory such that one can construct RNN architecture using simple layers and possibly any functions.

Firstly, a focus will be put on gradient-based algorithm (e.g. SGD) to train the nns. SGD will then be used as a baseline against which we will test gradient-free algorithms (e.g. Evolutionnary Algorithms).

Bonus

• [ ] static nn
• [ ] staticly trained nn

Research Goals

This library will be used to try to answer the following questions:

• Optimizations
  • How good are gradient free algorithms in training (r)nn ? Metrics: Training Time / Precision / Generalization
  • Benchmark Several different algorithms against a set of problems.
• Weights Augmentation
  • Gradient-free optimizations algorithms can be much more time consuming than SGD. We could remove this issue by applying dimensionality augmentation technics to generate the weights of the NN with a small intial weigths vector (This can be compared to Genetic Algorithm). Hence the goal would be to test different technics to augment small vectors.
  • Random Sampling
  • Random Matrix
  • Random NN
  • Random * with constraint (e.g. distribution of final weigths).
• Sparsification
  • Weights augmentation allow us to work with potentially huge neural networks (Billion of weigths). The resulting efficiency is still to be studied but what we can be sure of is that it will be energy consuming to use a huge RNN or FFNN on ligthweigth device (iot, mobile, ...). Hence we would like to come up with a way to reduce drastically the number of nonzero element in every matrices. There is several ways to do this. 1) Spectral sparsification (between learning phase ?). 2) Use the fact we used weigths augmentation. 3) Train sparse matrix (e.g. block matrices).
• Connection Optimization
  • The use of gradient-free optimization allow us to optimize the way nodes in the neural networks are connected. Discussion: We could either want to modify the connection of the neural network viewed as a graph or the connection between computational component of the nn.
• Scientific paper generation
  • Because why not ? And also this could be a nice POC that rnnlib works well for image generation (let's implement GANs).
• "Linear" Tropical Neural Networks
  • mult -> plus & plus -> max/min
  • The use of gradient free optimizations algorithms allow us to train Neural Network defined over a tropical algebra. This could be nice because the dot product becomes a non-linear function of the weigths.
  • Questions: Non-linear functions useful in this context ? Some theory possible ? Does the Universal Approximation Theorem stands here ? What can we do with a "Linear Tropical Neural Network" (only Matrix operation).
• Compression

TODO

• RNN

  • Matrices
  • Optimization
    • Use a single "dot" product and optimize it.
    • Matrix abstract multiplication: mult(auto override this T)
  • Vectors
  • Optimization
    • alias array for vector ?
    • use arrayfire ?
  • Layers
  • ~Linear~
  • ~Functional~
  • ~Recurrent~
  • Neural Network

• TRAINING

  • Evolutionary Algorithms
  • Evolution Strategy
  • Genetic Algorithm
  • Particle Swarm optimization
  • Ant colony optimization
  • Other Gradient-free Optimization
  • Nelder–Mead Simplex
  • DIRECT
  • DONE
  • Pattern Search
  • MCS

• TESTS

  • Functional Tests
  • Some good examples
  • Visualizable Tests
  • test the optimization algorithm for drawing graphs: force directed layout gives a function.
  • Machine Learning Tests
  • Adding Problem
  • XOR
  • Copying memory
  • Pixel-by-pixel MNIST (+ permuted)
  • NLP ?
  • Music Generation ?
  • Deep Reinforcement Learning Final Tests
  • Gym
  • Universe (too large a dependency)
  • Zbaghul

• GPU

  • OpenCL computing of graph
  • translate entire graph to openCL kernel?
  • optimization pass with similar system to https://github.com/CNugteren/cltune

• DOCUMENTATION

  • Vector
  • Matrix
  • Layer
  • Neural Network
  • Optimization Algorithms
  • EA
  • OGFO

• FORMAT

  • CodeCov
  • For most of these