search

TianhongDai / reinforcement-learning-algorithms

This repository contains most of pytorch implementation based classic deep reinforcement learning algorithms, including - DQN, DDQN, Dueling Network, DDPG, SAC, A2C, PPO, TRPO. (More algorithms are still in progress)
662 stars 109 forks source link

Plotted Reward Scale #9

Open OscarHuangWind opened 2 years ago

OscarHuangWind commented 2 years ago

Hi, I am oscar and I do appreciate those source codes with integrating various algorithms.

I have tried to run the nature DQN with default setting through Pong and BeamRider environment and found that the reward scale is not as large as the one posted in main page. For Pong Environment, I just manually set the clip_rewards = False and got the final mean around 27.430 which is far from the max level(around 300) posted.

Is it due to difference hyper-parameters setting or may be due to some plotting techniques? BTW, I will really appreciate if you can update the plotting code, Thank you!

Screenshot from 2021-10-27 13-15-38

TianhongDai commented 2 years ago

@OscarHuangWind Hi Oscar - Thank you for pointing out this problem. This is because of the setting of the environment:episode_life. The reward displayed in the terminal is not the real accumulated reward, the real accumulated reward is recorded in the logs/<envs> folder, and you need to use those files to plot the learning curve. Could you please check the file in that folder firstly?

In addition - I will upload / update the plot script. If you have further questions, please let me know.

OscarHuangWind commented 2 years ago

Hi, thank you for your quick reply. I plotted .csv file in "logs/envs" and found that the reward scale is right. But still I don't understand the reason.

Currently I am doing research in DQN algorithms with multiple atari environments and suffered from the reward scale problem as well. Would you mind to write me an email and discuss more specifically? Or I will appreciate if you leave me an email address.

My email address is: wenhui001@e.ntu.edu.sg Thank you in advance!

TianhongDai commented 2 years ago

@OscarHuangWind Hi Oscar, No Problem, I will drop you an email today.