Open rajfly opened 1 year ago
Update: After some testing, I found out that the problem lies with the custom env function. After changing the env from a custom function to simply using the ones declared in Ray's registry with Rays preprocessing, the PPO training results seems to be ok. Though, I am not sure why this is so. Perhaps there is a bug with passing a custom env fn.
def env_creator(env_config):
env = gym.make(f"ALE/{config.env}-v5", frameskip=1, render_mode='rgb_array')
env = AtariPreprocessing(env, noop_max=0, scale_obs=True)
env = TransformReward(env, lambda x: np.clip(x, -1, 1))
env = FrameStack(env, 4)
return env
As you noted, this is different from what RLlib does internally.
RLlib has a range of fine-tuned settings under the tuned_examples
folder.
What happens if you use one of these (ray/rllib/tuned_examples/ppo/atari-ppo.yaml
) and apply it to your problem, how are the results then?
What happened + What you expected to happen
Hi, I recently tried to recreate the experiments from the original PPO paper. First I used Stable Baselines3 to do so and noticed that the reward generated for the Alien atari environment was close to what was shown on the paper. I utilised similar hyperparamters that were used in the original paper and the results did not deviate that much with a mean reward of more than 1500.
I recently tried to do the same with RLlib instead of stable baselines3. Using the same hyperparameter set, I tried to train the PPO algorithm on the Alien atari environment. I however only noticed a reward of not more than 100 at the end. I am not sure what went wrong during the training process and was hoping someone would be able to help me out.
Below is the list of intended hyperparameters which closely follow the original PPO paper. Using this hyperparameters Stable Baselines3 was able to train the Alien environment till a reward of more than 1500 after 10M timesteps. RLlib however struggled to even attain a reward of more than 100 after 10M timesteps:
Environment:
Networks:
Algorithm:
I have already posted this issue on https://discuss.ray.io but it got temporarily hidden by a spam filter. My username for discuss.ray.io is rajfly. If there are any admins out there, please allow my post. Thanks.
Versions / Dependencies
RLlib version: 2.4.0 (all my pervious experiments were with this version so to keep things consistent I am using the same version). I however also tested on the latest RLlib version and noticed the same issues. Python version = 3.9 OS = Ubuntu LTS
Reproduction script
This is the reproduction script, to run simply execute as such:
python file_name.py --gpu 0 --env Alien --algo ppo
Issue Severity
High: It blocks me from completing my task.