Implement a Genetic algorithm optimizer to find Weights and Biases faster for large complex neural networks

[jaykay9999]

If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with the documentation (for small docs fixes please send a PR instead). The form below must be filled out.

Here's why we have that policy:.

Keras developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information.

TensorFlow version (you are using): Latest Are you willing to contribute it (Yes/No) : Yes

Describe the feature and the current behavior/state.

Describe the feature clearly here. Be sure to convey here why the requested feature is needed. Any brief description about the use-case would help.

I have been working with both Neural networks and Genetic algorithms for a long time, and I thought why not mix both. So the idea is an optimizer that uses GA to find the best weights and parameters through initializing multiple neural nets with random weights and biases, then the ones with the best accuracy will survive, and repredouce (crossover) to give the second generation of neural nets in which (a child neural net has 50% of the each parent weights and biases) and it keeps going on. Here is an article that I found online from someone else that also shares the same idea : https://towardsdatascience.com/using-genetic-algorithms-to-train-neural-networks-b5ffe0d51321#:~:text=What%20are%20genetic%20algorithms%3F,in%20a%20better%20neural%20network.

I also asked GPT-4 about this idea and here is what it responded :

Me Jasem :

iam currently thinking of implementing an optimizer called genetic algorithms to keras tensorflow library in which a population of random weights and biases will initialized, (say 1000 diferent neural networks with different random weights and biases), then the ones with the best accuracy will be added to a mating pool to crossover and give the second generation of 1000 neural networks. do you think is would be of great value for keras if done right ? i found articles online speaking about it

GPT4 :

Genetic Algorithms (GAs) are a powerful class of optimization methods based on the principles of evolution and natural selection. They can be quite effective in solving optimization problems where the search space is large, complex, or poorly understood. In the context of neural networks, GAs can be used to optimize the weights and biases, the structure of the network itself, or both. The approach you're describing, where a population of random weights and biases are initialized and then evolved over time, is known as neuroevolution.

There's a lot of potential value in integrating a genetic algorithm-based optimizer into a popular machine learning library like Keras/TensorFlow:

Flexibility: GAs are quite versatile and can be applied to a variety of optimization problems, including those where gradient information is not available, which is a limitation of traditional backpropagation-based training methods.

Novelty: While other evolution-inspired optimization methods like evolutionary strategies and particle swarm optimization are already in use in machine learning, full-fledged GAs are less common, so this could be an interesting and novel contribution.

Potential for improved performance: GAs can potentially find better solutions than traditional optimization methods, especially for complex tasks or tasks where the optimal network architecture is not known in advance.

Parallelizability: GAs are inherently parallelizable, as each individual in the population can be evaluated independently. This could make your approach well-suited to modern hardware architectures.

However, there are also some challenges and potential downsides to consider:

Computational cost: GAs can be computationally intensive, especially if the population size is large. This could make them impractical for large-scale problems or for use on limited hardware.

Convergence: While GAs can potentially find very good solutions, they may not always converge to the global optimum, especially for complex optimization problems.

Parameter tuning: GAs have a number of parameters (population size, mutation rate, crossover rate, selection method, etc.) that need to be tuned, and getting these settings right can be tricky.

Complexity: Implementing a GA-based optimizer in a way that's easy to use and integrate with existing code could be challenging.

In conclusion, integrating a GA-based optimizer into Keras/TensorFlow could certainly be valuable and would likely be of interest to the machine learning community. However, it would also be a significant undertaking and would require careful design and implementation to ensure that the benefits outweigh the potential downsides.

---

The very difficult part here is making a great crossover function that ensures accuracy is increasing from a generation to another.

Will this change the current api? How?

No, It would be called the same way softmax, or any other optimzier is being called.

Who will benefit from this feature?

Keras users who are implementing very large neural networks with relatively low computing power.

Contributing

• Do you want to contribute a PR? (yes/no): Yes, All I need is a confirmation from someone who is not GPT4
• If yes, please read this page for instructions
• Briefly describe your candidate solution(if contributing): My candidate solution would be an optimizer that users can call to find weights and parameters potentially much faster when dealing with a large number of neurons and data. Based on my experience with Genetic algorithms, a simple crossover function will not be of great use here, so I consider it as the only challenge for this feature. I wish i spoke about this in the Keras community meeting yesterday directly with the attendees to discuss it in details, but if you have any questions or considerations please let me know.

[tilakrayal]

@jaykay9999, Could you please elaborate about your Feature. Also, please specify the Use Cases for this feature. Thank you!

[jaykay9999]

@tilakrayal sure, I will add the genetic algorithm optimizer in my local fork and test it compared to 3 other optimizers. is there a specific test or something that i should know (template or format) or just pass travis CI tests ?

[jaykay9999]

I designed a diagram that shows how the genetic algorithm works as an optimizer in a very small neural network with very small size of population individuals. (it is the same logic used when breeding dogs assuming the end goal is having a very very tiny dog breed over generations)

Assume an initial population of 3 lists of random weights & biases. they will be used by the neural network one by one to calculate the error using each.

Then, the ones with the lowest error will join a matin pool to crossover and produce childs that will have some weights & biases from parent 1 and parent 2 for each child. this is called natural selection.

Finally, Generation 2 is produced, which is a population that is a descendant of the fittest individuals of generation 1. there is also a slight chance for a child to have a random mutation (same as learning rate) to keep space for finding other patterns.

then repeat over and over and over.

In reality, you would start with a population of 20000 dna's, each dna is a list of weights and biases that is equal to the number of neurons in the network. the end goal is to optimzie the error generation by generation.

This is useful when either the data is complex or large (requires too much computing power). Genetic algorithms are search optimization algorithms often used to find shortest paths very quickly.

Note that there are multiple ways to implement the crossover function which is very crucial here since it decides if the next generation average error will be less than the previous one.

I can give it a shot and show you results. If you have further questions or suggestions please let me know :)

[jaykay9999]

I can explain the idea verbally through google meets or any method if you would like to, in case the details i provided are not that clear in terms of possibilities and complexity.

[gowthamkpr]

Hi @jaykay9999, thank you for the idea but is there a research paper that discusses about concrete implementation of this idea because keras team thinks there should be a significant interest in user community before they can invest resources into this. Hope this helps. Thanks!!

[jaykay9999]

Hi @gowthamkpr

I can go for a concrete implementation with documentation if you would like to, all I need to know is if there is a specific data or specific architecture that i should try to test an optimizer. the idea i'am proposing can be a research itself due to the large combination of possibilities to develop this (there is also another possbility where the architecture of the neural net is decided by a genetic algorithm rather than the weights and biases - neuroevolution).

I'am already conducting research right now in the field of software engineering and I have a clear idea on how I can create good research. If you would like me to invest in this I would appreciate it. I tested genetic algos on multiple fields (timetable optimization and scheduling, trading...) and I will give this a shot either way in my local fork and test it. If you would like to discuss anything about this I'am open for any email or call :)

jasemkhlifi@gmail.com

[jaykay9999]

here are some interesting research i found about this idea on google scholar

https://link.springer.com/article/10.1007/s10462-011-9208-z An optimizing BP neural network algorithm based on genetic algorithm Shifei Ding, Chunyang Su, Junzhao Yu Artificial intelligence review 36, 153-162, 2011

https://www.sciencedirect.com/science/article/pii/016781919090086O Genetic algorithms and neural networks: Optimizing connections and connectivity Darrell Whitley, Timothy Starkweather, Christopher Bogart Parallel computing 14 (3), 347-361, 1990

https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=2d02fb2a733efde95e008a0ecbe1aed09d851152 Optimizing weights of artificial neural networks using genetic algorithms Kinjal Jadav, Mahesh Panchal Int J Adv Res Comput Sci Electron Eng 1 (10), 47-51, 2012

if you want more you can search the term on google scholar (there are different approaches) : optimizing neural network weights with genetic algorithms

[fchollet]

Thanks for the suggestion. Interesting idea, but it doesn't make sense to add this to Keras. If you'd like to experiment with neuroevolution (which can be done with a Keras model as a starting point), you can take a look at existing experimental libraries, such as nevopy. https://github.com/Talendar/nevopy

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No