Closed sdpkjc closed 1 year ago
A problem was reported in #259 , and the problem appears to exist in ddpg(ddpg_continuous_action.py & ddpg_continuous_action_jax.py) as well.
ddpg_continuous_action.py
ddpg_continuous_action_jax.py
the action bias is added twice to the action. First, it is added during actor forward pass. Second, the random noise comes from a distribution with the center in actor.action_bias.
# In ddpg_continuous_action.py line176 # https://github.com/vwxyzjn/cleanrl/blob/17febbf88c28d3572c62ae6fa041921ef84290c4/cleanrl/ddpg_continuous_action.py#L176 actions += torch.normal(actor.action_bias, actor.action_scale * args.exploration_noise) # In ddpg_continuous_action_jax.py line232 # (https://github.com/vwxyzjn/cleanrl/blob/17febbf88c28d3572c62ae6fa041921ef84290c4/cleanrl/ddpg_continuous_action_jax.py#L232) jax.device_get(actions)[0] + np.random.normal(action_bias, action_scale * args.exploration_noise)[0]
poetry install
actions += torch.normal(actor.action_bias, actor.action_scale * args.exploration_noise)
actions += torch.normal(0, actor.action_scale * args.exploration_noise)
jax.device_get(actions)[0] + np.random.normal(action_bias, action_scale * args.exploration_noise)[0]
jax.device_get(actions)[0] + np.random.normal(0, action_scale * args.exploration_noise)[0]
Problem Description
A problem was reported in #259 , and the problem appears to exist in ddpg(
ddpg_continuous_action.py
&ddpg_continuous_action_jax.py
) as well.Checklist
poetry install
(see CleanRL's installation guideline.Possible Solution
actions += torch.normal(actor.action_bias, actor.action_scale * args.exploration_noise)
➡️actions += torch.normal(0, actor.action_scale * args.exploration_noise)
jax.device_get(actions)[0] + np.random.normal(action_bias, action_scale * args.exploration_noise)[0]
➡️jax.device_get(actions)[0] + np.random.normal(0, action_scale * args.exploration_noise)[0]