search

uber-research / deep-neuroevolution

Deep Neuroevolution
Other
1.63k stars 298 forks source link

ranking is not done correctly #20

Closed adam-katona closed 5 years ago

adam-katona commented 5 years ago

In ES, compute_ranks() does an argsort, which will give different ranks to individuals with the same fitness. This introduces a noise in the gradient estimate. This is not a big issue since the expected value of the noise is zero, but it can slow down convergence. This is really only a problem in environments, where rewards are sparse, so a lot of individuals will have the same fitness. Solution: Average ranks for individuals with equal fitness.

adam-katona commented 5 years ago

I just realized, that this might be intentional, so there is some extra exploration in flat regions of the fitness function.