[rllib] ARS optimizer differs from paper

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

https://ray.io

Apache License 2.0

33.11k stars 5.6k forks source link

[rllib] ARS optimizer differs from paper #2909

Closed whikwon closed 5 years ago

whikwon commented 6 years ago

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
Ray installed from (source or binary): source
Ray version: 0.5.2
Python version: 3.6.3
Exact command to reproduce:

Describe the problem

In the ars.py code, the model use Adam optimizer for training. When I looked into the paper, SGD was used for training.

Is there any reason or experimental result for using Adam rather than SGD?

Source code / logs

ericl commented 6 years ago

I believe this code is directly adapted from the reference implemention cc @eugenevinitsky

eugenevinitsky commented 6 years ago

This is a mistake; thank you for catching it. The reference code used SGD as you pointed out, but Evolutionary Strategies used ADAM and that wound up getting copied over. I will push a fix shortly.

eugenevinitsky commented 6 years ago

@whikwon PR for this will be up shortly