search

CMA-ES / libcmaes

libcmaes is a multithreaded C++11 library with Python bindings for high performance blackbox stochastic optimization using the CMA-ES algorithm for Covariance Matrix Adaptation Evolution Strategy
Other
321 stars 78 forks source link

Examples for pull request #139 #141

Closed montanier closed 3 years ago

montanier commented 8 years ago

This is an example that requires the modifications proposed in the pull requesti #139. The intend is to find an example as minimal as possible.

The scenario is as follow: a car is in a 1D world and needs to move as far as possible. The controller of the car can perceive the slope behind the car and in front of the car. The controller has one output that is either up (+1) or down (-1). For the car to move forward, the value of the output should match the direction of the slope.

The important points are that the world is random and that the position of the car is never reset. Therefore, the optimization is noisy and online. The online part of the optimization requires the modifications of pull request #139.

The car is controlled by an extremely simplified neural networks : just two weights linking the two inputs to the output. The parameters of the neural network are optimized by CMA-ES. The ideal output should be 0 for the first weight, and more than 1 in the second weight. CMA-ES is able to find these weights.

beniz commented 8 years ago

Thanks @montanier very useful example I believe. Quick question: when running the sample_code_online exe, the output are the two weights you are mentioning, and the first weight is not always close to 0. Does this mean the optimization fails to find the proper values ? If yes, is there a setting you know of that would allow to find the best weights all (if not most of) the time ? I tried bumping evaluationTime to 1000 with no change in the convergence. The reason for this is to try to keep very reproducible results in the examples repository so that it is obvious when something is off. That said, no hard requirement for a merge IMO, just a bonus if available easily.

montanier commented 8 years ago

Unfortunately, i do not have a quick fix for this problem at the moment. Basically, as long as the second weight is much higher than the first it will workout. Having a first weight a bit away from 0 is not idea but can workout. I can think of something around limiting the range of weights or penalizing weights when they are too high.

I understand that it would be much better to be able to write a test for that. I will look at how to improve things a bit in the test scenario. Do you have a list of tests for the examples for folder ? I could add the test myself there.