[TODO] More optimizers

[pluskid]

Currently we have SGD and Adam. Should be relatively simple to add more, like RMSProp, etc. Some references:

• http://cs231n.github.io/neural-networks-3/#update
• https://github.com/Lasagne/Lasagne/blob/master/lasagne/updates.py
• https://github.com/fchollet/keras/blob/master/keras/optimizers.py

[tbreloff]

I would consider adding AdaMax as well... it's very briefly referenced in the Adam paper, but I think it's actually better.

[pluskid]

@tbreloff Yes that would be great! I somehow get swamped in something else and was not able to add those shortly. But you are very welcome to contribute AdaMax (or other optimizers that you think is useful).

[mikewl]

Is there interest in a Genetic Algorithm backend? I can start a pure Julia one once I have played around with MXNet more. I don't have experience however with adding the CUDA backend.

[vchuravy]

@Mike43110 Optimizers don't require CUDA backends. What kind of genetic algorithm would you like to add?

[mikewl]

I am planning on starting with just a basic one. Will have elitism as a flag and multiple crossover and mutation types. Will then move on to trying an adaptive GA and parallel implementations if performance is improved. My previous tests did not however show significant improvement with processor parallelism. Threading at a later stage along with testing of shared arrays still needs to be done.

I have not attempted using a PSO or DE based evolution of a network yet. But, I should be able to add those as well.